Efficient, dynamic, high contrast lensing with applications to imaging, illumination and projection

ABSTRACT

A new projector design combines one spatial light modulator that affects only the phase of the illumination, and one spatial light modulator that only affects its amplitude (intensity). The phase-only modulator curves the wavefront of light and acts as a pre-modulator for a conventional amplitude modulator. This approach works with both white light and laser illumination, generating a coarse image representation efficiently, thus enabling, within a single image frame, significantly elevated highlights as well as darker black levels while reducing the overall light source power requirements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/981,069 filed 16 May 2018, which is a continuation of U.S.application Ser. No. 15/368,021 filed 2 Dec. 2016 now issued as U.S.patent Ser. No. 10/003,776, which is a continuation of PCT InternationalApplication No. PCT/CA2015/050515 filed 3 Jun. 2015. PCT InternationalApplication No. PCT/CA2015/050515 claims priority from U.S. ApplicationNo. 62/007,341 filed 3 Jun. 2014 and U.S. Application No. 62/118,945filed 20 Feb. 2015. For purposes of the United States, this applicationclaims the benefit under 35 U.S.C. § 119 of U.S. Application No.62/007,341 filed 3 Jun. 2014 entitled DYNAMIC FREEFORM LENSING WITHAPPLICATIONS TO HIGH DYNAMIC RANGE PROJECTION and U.S. Application No.62/118,945 filed 20 Feb. 2015 entitled EFFICIENT, NUMERICAL APPROACHESFOR HIGH CONTRAST FREEFORM LENSING, both which are hereby incorporatedherein by reference for all purposes.

TECHNICAL FIELD

This invention relates to generating desired patterns of light. In someembodiments the desired patterns of light correspond to images specifiedby image data. Specific embodiments provide methods for controlling afree-form lens such as a phase-shifting light modulator, a variablemirror, or the like to achieve a desired distribution of light. Otherembodiments provide projectors for projecting light.

BACKGROUND

Both light efficiency and dynamic range are major concerns forcommercial projector designs. High contrast and peak luminance are vitalfor higher perceived image quality (brightness, colorfulness) [Rempel etal. 2009], even if most images only require a small amount of localizedvery bright highlights above their average picture level in order toappear realistic [Rempel et al. 2011]. On the other hand, an opticalsystem should be highly efficient to minimize power consumption, andsimplify thermal management. The latter concern makes it impractical toachieve very high peak brightness by boosting the power of a projectorlight source.

Amplitude spatial light modulators (or SLMs) are often used to createtones and colors in images by pixel-selectively blocking light. SuchSLMs tend to be optically inefficient since blocked light is absorbed.

HDR (high dynamic range) image projection may be achieved by providingtwo or more stages of light modulators (Hoskinson et al.). Many lightmodulators (e.g. LCD panels) generate a desired light field bysubtraction (i.e. by absorbing unwanted light). Some efforts have beenmade to create desired light fields by reallocating light. However, manyavailable light reallocation technologies have significantdisadvantages. For example, some require laser light, which can resultin laser speckle. Some are very computationally intensive. Some requirevery high spatial frequency control of light which places demands onlight modulators and also can result in artifacts caused by diffractionof light.

Freeform lenses, which can be aspherical, asymmetric lenses may bedesigned to generate specific caustic images under pre-definedillumination conditions [Finckh et al. 2010, Papa et al. 2011,Schwartzburg et al. 2014, Yue et al. 2014]. The caustic image is aredistribution or “reallocation” of light incident on the freeform lens[Hoskinson et al. 2010], Computer graphics approaches to designing suchfreeform lenses are known as goal-based caustics. Designing a freeformlens to achieve a particular desired image can be computationallyintensive.

Freeform lenses may be applied for general lighting applications (e.g.[Minano et al. 2009]) and more specifically for goal-based caustics[Berry 2006, Hullin et al. 2013]. Some methods for designing freeformlenses apply discrete optimization methods that work on a pixelatedversion of the problem (e.g. [Papas et al. 2011, Papas et al. 2012,Papas et al. 2012]). Others optimize for continuous surfaces withoutobvious pixel structures (e.g. [Finckh et al. 2010, Kiser et al. 2013,Pauly and Kiser 2012, Schwartzburg et al. 2014, Yue et al. 2014]).

Holographic image formation models (e.g. [Lesem et al. 1969]) have beenadapted to create digital holograms [Haugen et al. 1983]. Holographicprojection systems have been proposed for research and specialtyapplications [Buckley 2008]. Many of these systems use diffractionpatterns (or holograms) addressed on a phase SLMs in combination withcoherent light (lasers) for image generation. While in principle anefficient way to form an image, the challenges in holography forprojectors lie in achieving sufficiently good image quality, the limiteddiffraction efficiency achievable by binary phase modulators [Buckley2008], and the requirement for a Fourier lens, often resulting in abright DC spot within the active image area or reduced contrastthroughout the image due to an elevated black level (in cases where theDC spot is expanded). Holographic projection generally requires coherentlight.

The inventors have recognized a need for more efficient ways to designfreeform lenses to achieve desired light patterns. In particular, theinventors have determined that sufficiently efficient design methods maybe applied to provide real-time or near real time generation of dynamicfreeform lenses. Such dynamic freeform lenses may, for example delivervideo content or dynamically-changing light effects.

SUMMARY

This invention provides methods for controlling spatial light modulatorsto provide free-form lensing of light. The light may be projected and/orfurther modulated. Another aspect of the invention provides apparatussuch as projectors, displays, illumination systems and their componentsthat implement methods as described herein.

Dynamic freeform lenses may be applied in light projection systems. Suchlight projection systems may advantageously be light efficient, providehigh (local) peak luminance, and high contrast (high dynamic range,HDR). Some embodiments employ a dynamic freeform lens, implemented on aphase only SLM. The phase only SLM may be combined with a conventionallight blocking SLM such as a reflective LCD in a cascaded modulationapproach. When controlled as described herein a phase modulator cancreate a smooth, but still quite detailed “caustic” image. Such acaustic image may be further modulated by an amplitude modulator if sodesired. This approach may provide both a higher dynamic range and/orimproved (local) peak luminance as compared to conventional projectors.

This application describes inter alia:

-   -   illumination systems and projectors in which a phase modulator        is illuminated with (near-)collimated light and a phase pattern        addressed on the phase modulator forms an image or desired light        field with or without further optical elements;    -   a Fourier domain optimization approach for generating freeform        lens configurations that is capable of high frame rates for        dynamic light steering using phase modulators;    -   real time freeform lensing algorithms and their applications in        illumination systems, projectors and video/image processing        systems;    -   a dual-modulation projector design that combines a phase        modulator and an amplitude modulator for image generation and is        capable of working with broadband light as well as monochromatic        light (such as laser light).

An example freeform lens optimization approach is based on first-order(paraxial) approximations, which hold for long focal lengths and arewidely used in optics. Under this linear model, the local deflection oflight is proportional to the gradient of a phase modulation function,while the intensity is proportional to the Laplacian. The phasemodulation function can be solved for in the in the lens plane insteadof the image plane, for example using optimization methods, to arrive ata very simple to implement method that optimizes directly for the phasefunction or the shape of a refractive lens, without requiring additionalsteps. This approach may be solved very efficiently in the Fourierdomain. In some embodiments the algorithm is efficient enough for on-thefly computation of freeform lensing configurations for reproducing videosequences.

One example aspect provides a dual-modulation projector design, in whichone spatial light modulator that affects only the phase of theillumination is combined with one spatial light modulator that affectsits amplitude (intensity). The phase-only modulator curves the wavefrontof light reflected off it, and acts as a pre-modulator for aconventional amplitude modulator. This approach works with both whitelight and laser illumination, generating a coarse image representationwithout significant loss of energy.

The dual-modulation HDR projector design uses the freeform lensoptimization approach to provide energy efficient high dynamic range andhigh intensity projection. This approach is capable of using white light(or other broadband light) illumination as well as coherent laser light.Use of broadband light can yield a significant improvement in imagequality by eliminating laser speckle and averaging out other diffractionartifacts. A real-time implementation of a high resolution freeform lensenables applications such as video processing. A dual-modulation HDRprojector may be constructed entirely from robust components that arecurrently commercially available.

In some embodiments the phase modulator creates a smoothed, but stillquite detailed “caustic” image on the amplitude modulator. Since thecaustic image merely redistributes, or “reallocates”, light thisapproach produces both a higher dynamic range as well as an improved(local) peak brightness, compared to conventional projectors thatmodulate light using a single amplitude modulator.

Some embodiments apply a linear model in which the local deflection oflight is proportional to the gradient of a phase modulation function,while the intensity is proportional to the Laplacian.

Some embodiments combine application of this model with aparameterization of the optimization problem in the lens plane insteadof the image plane to arrive at a very simple to implement method thatoptimizes directly for the phase function or the shape of a refractivelens, without any additional steps. Although the objective function isnon-convex due to an image warping operator convergence can typically beachieved within a few iterations.

Technology as described herein has application in controlling dynamicfreeform lenses, for example in the context of light efficient, high(local) peak brightness, and high contrast (high dynamic range, HDR)projection systems.

Some aspects of the invention provide algorithms which may be applied toefficiently determine phase patterns for a phase modulator to cause adesired light profile in the image plane. In some embodiments a (near)one-to-one relationship is established between the phase at a locationin the lens plane and a corresponding area of the image plane. This isin contrast to the diverging or converging rays or beams that arerequired for traditional holographic approaches.

Further aspects and example embodiments are illustrated in theaccompanying drawings and/or described in the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting example embodiments ofthe invention.

FIG. 1 is a schematic illustration of an example geometry for imageformation. Phase modulation takes place in the lens plane, which isplaced at a focal distance of f from the image plane. This results in acurvature of the wavefront, represented by a phase function p(x).

FIG. 2 is a schematic illustration showing intensity change due to thedistortion of a differential area dx.

FIG. 3 is a schematic illustration showing a geometry for refraction ina freeform lens defined by a height field h(x).

FIG. 4 shows stages in an algorithm for freeform lensing.

FIGS. 5C, 5D, and 5E show examples of refractive lenses produced usingmethods described herein.

FIGS. 5A and 5B show a phase-only spatial light modulator being used todrive a projector display with white light. The same setup could alsouse laser illumination. This approach is particularly useful inenergy-efficient dual modulation HDR projectors. The right hand imageshows refractive lenses designed using the same free form lensingalgorithm for goal-based caustics. For photography purposes, bothresults are shown on back-illuminated rather than front screens, so thatthe displayed ‘Lena’ image appears mirrored.

FIGS. 6A and 6B are photographs of prototype embodiments. Layout of anarrowband, dynamic lensing test setup comprising a HeNe laser source, abeam expander, a linear polarization filter and folding mirrors, thephase-only SLM and a projection screen at 50 mm distance from the SLM.The SLM phase pattern used to generate the freeform lens (in this casethe Siggraph logo) is also displayed on the notebook screen forvisualization. Note the Fresnel-like phase wrapping used to achievelarger phase changes. Bottom: the white light configuration bypasses thelaser module, and comprises a white LED, collimation optics and linearpolarization filter, the phase-only SLM and a projection screen at 50 mmdistance from the SLM. The SLM in this setup was calibrated for a centerwavelength of 550 nm.

FIG. 7 is a system diagram of an example high brightness, HDR projector:light from an expanded and collimated laser beam is reflected off aphase-only modulator. The per-pixel amount of phase retardationresembles the height field of the dynamic lens calculated with analgorithm as described herein. The effective focal plane of this freeform lens is in-plane with an off-the-shelf, reflective projection headconsisting of the polarizing beam splitter together with an LCoSmicrodisplay and a projection lens. Light from dark parts of the imagecan be used to create high luminance features, and simultaneously reducethe black level.

FIG. 8 is a system diagram of an example high brightness, HDR projectorincluding an intermediary image plane in which light from the phasestage can be further shaped, for example by adding a light shapingdiffuser: light from an expanded and collimated laser beam is reflectedoff a phase-only modulator. The per-pixel amount of phase retardationresembles the height field of the dynamic lens calculated with analgorithm as described herein. The effective focal plane of this freeform lens is in-plane with an intermediary image plane, which is relayedonto an off-the-shelf, reflective projection head comprising thepolarizing beam splitter together with an LCoS microdisplay and aprojection lens via relay optics. Light from dark parts of the image canbe used to create high luminance features, and simultaneously reduce theblack level.

FIG. 9 shows the comparison of simulated and captured results from topto bottom by row. Phase Pattern: the phase pattern as computed byAlgorithm 1. Simulation: Huygens-Fresnel simulation of predicted image.Direct: photograph of actual image without diffuser showing diffractionartifacts. Diffuser: by adding the a thin-film diffuser, artifacts suchas diffraction fringes nearly completely mitigated. Standard: photo ofstandard, amplitude modulation only projection using a single amplitudemodulator shows elevated black levels and low contrast. Proposed (HDR):Using our lensing approach redistributes light from dark regions tobright regions, resulting in improved black levels and increasedhighlight intensity. The last two rows appear slightly distorted due toan off-angle position of the camera which became necessary because of ashort throw projection and close screen as well as baffles to blockambient light effectively to capture the black level of the system.

FIGS. 10A, 10B, and 100: From left to right correlating to positions Ato C in FIG. 8: A: phase pattern present at phase-only LCoS modulator,B: a direct image produced by lens in intermediary image plane (prior todiffuser) and C: intensity distribution present at amplitude LCoSmodulator after having passed through a thin-film light-shapingdiffuser.

FIGS. 11A, 11B, and 11C show an example high-dynamic range projectionsystem based on dual modulation. A first stage modulates the sourceillumination phase to form a coarse intermediate image. This is followedby an amplitude modulation stage that forms the final image. Using phasemodulation results in greater contrast and darker black-levels thanconventional projection since light is redistributed rather thanblocked.

FIG. 12A shows geometry for the image formation model, with phasemodulation p(x) taking place in the lens plane, and resultingdeflections creating a caustic image on the image plane at distance f.FIG. 12B shows the local intensity on the image plane is related to thechange in the differential surface area between corresponding patches onthe lens plane and the image plane.

FIGS. 13A, 13B, and 13C: By mirror-padding the input image, pure-Neumannboundary conditions at the image edge can be achieved while retaining aToeplitz matrix structure. This prevents distortions of the imageboundary. Simulated results with LuxRender™.

FIGS. 14A, 14B, 14C, and 14D: LuxRender raytracing simulations: thesmoothness parameter a penalizes strong caustics in the image thatachieve high-brightness but poor image quality.

FIG. 15: Layout of a simple example dynamic lensing test setup for usewith broadband light. A beam of light from a light source such as awhite LED with collimation optics (a modified flash light) together witha linear polarization filter (provided for sensible use of the phasemodulator) is reflected off the SLM operated in phase-only mode and ontoa small projection screen facing the SLM at a 50 mm distance. The SLM inthis setup was calibrated for a center wavelength of 550 nm. Due tolight-engine power limitations, this setup was not sufficient to drive adual-modulation setup (the reduced intensity also introduces cameracapture noise in the inlay) although it illustrates that phasemodulation is functional with broadband light. This paves the way forfuture broadband illumination phase+amplitude dual-modulation setups.Such setups could apply industry standard Xenon bulbs, cost effectiveblue laser+phosphor light sources or LEDs, for example as light sources.

FIG. 16: Single modulation test setup for lasers comprising a lightsource (yellow box, 532 nm DPSS laser and laser controller), beamexpansion and collimation optics (orange box), the reflective phase SLM(blue), various folding mirrors and a simple projection lens to relaythe image from and intermediate image plane onto the projection screen(green). The phase pattern shown on the computer screen correlateslinearly to the desired phase retardation in the optical path to formthe image. It has been phase-wrapped at multiples of one wavelength andcan be addressed directly onto the micro display SLM.

FIG. 17: Simplified system diagram of an example high brightness, HDRprojector: light from an expanded and collimated laser beam is reflectedoff a phase-only modulator. The per-pixel amount of phase retardationresembles the height field of the dynamic lens calculated with ouralgorithm. The effective focal plane of this freeform lens is in-planewith an off-the-shelf, reflective projection head consisting of thepolarizing beam splitter together with an LCoS microdisplay and aprojection lens. Light from dark parts of the image can be used tocreate high luminance features, and simultaneously reduce the blacklevel.

FIG. 18: Comparison of simulated and captured results from top to bottomby row. Phase Pattern: the phase pattern as computed by Algorithm 4.1.Simulation: Huygens-Fresnel simulation of predicted image. Direct:photograph of actual image without diffuser showing diffractionartifacts. Diffuser: by adding the a thin-film diffuser, artifacts suchas diffraction fringes nearly completely mitigated. Standard: photo ofstandard, amplitude modulation only projection using a single amplitudemodulator shows elevated black levels and low contrast. Proposed (HDR):Using our lensing approach redistributes light from dark regions tobright regions, resulting in improved black levels and increasedhighlight intensity. The last two rows appear slightly distorted due toan off-angle position of the camera which became necessary because of ashort throw projection and close screen as well as baffles to blockambient light effectively to capture the black level of the system.

FIGS. 19A and 19B: Photos of a prototype projector in LDR comparisonmode (left image) and HDR mode (right image). Left: light redistributionis active resulting in increased peak luminance and reduced black level.Right: LDR projector for comparison using the same hardware. In LDR modea flat phase profile results in a uniform illumination profile at theamplitude attenuator (second SLM). Each image is vertically split toshow a long exposure time on the left half (dark level detail isvisible) and a short exposure on the right side (detail in thehighlights is visible). Both exposures are of the same projected imageon screen.

DETAILED DESCRIPTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive sense.

Freeform Lensing

Some embodiments provide a new approach to determining a lens shape orphase function that can provide a desired light field when illuminated.The output of this approach may be applied to control a phase modulatoror variable lens or variable mirror to yield the desired light field.

In displays according to some embodiments, phase-only SLMs are used asprogrammable freeform lenses. The lenses may be illuminated withbroadband light (e.g. white light). This eliminates speckle, while atthe same time the spatial smoothness of the lens modulation patternsreduces diffraction artifacts. Any remaining diffraction is averaged outby the broadband nature of the illumination, resulting only in a smallamount of blur that can be modeled and compensated for in adual-modulation setting

Some embodiments optimize directly for the phase function, or,equivalently, the lens shape, without a need for a subsequentintegration step. This is facilitated by a parameterization of theproblem that expresses the optimization directly in the lens planerather than the image plane. This leads to a much simpler formulation ofthe freeform lens optimization problem than the approaches described inthe literature.

Phase Modulation Image Formation

This application relates in part to methods for displaying desired lightpatterns by using a modulator that does not absorb much light, but movesit around within the image plane. In this way, light can be reallocatedfrom dark image regions to bright ones. For example the modulator may becontrolled to provide moving, bright spots of light. An example of amodulator suitable for this application is a LCoS SLM operated in aphase-only fashion. The SLM may have a suitable resolution such as 1, 25 or more megapixels. Control of the SLM may be achieved by optimizing acontinuous phase function representing the required curvature of thewavefront of light as it passes through the SLM.

Apparatus and methods according to different embodiments allow the useof broadband light (e.g. from a lamp, LEDs, or arrays of lasers withdifferent wavelengths) as well as monochromatic laser light. Phasemodulating arrays such as liquid crystal-based SLMs operated in aphase-only configuration are applied as programmable freeform lenses.Being able to use broadband illumination can help to eliminate screenspeckle, while at the same time the spatial smoothness of the lensmodulation patterns reduces other artifacts such as diffraction. Anyremaining diffraction effects in the image plane can be averaged out bythe broadband nature of the illumination, resulting only in a smallamount of blur that can be easily modeled and compensated for byproviding one or more additional modulators.

One way to optimize directly for the phase function (i.e. the shape ofthe wavefront in the lens plane), or, equivalently, the lens shape,without a need for a subsequent integration step involves aparameterization of the problem that allows us to express theoptimization directly in the lens plane rather than the image plane.

To derive the image formation model for a phase modulation display, weconsider the geometric configuration shown in FIG. 1: a lens plane andan image plane (e.g. a screen) are placed parallel to each other atfocal distance f. Collimated light is incident at the lens plane fromthe normal direction. A phase modulator (or lens) in the lens planedistorts the phase of the light, resulting in a curved phase functionp(x), which corresponds to a local deflection of the light rays. In arelated embodiment, a variable mirror is provided in the lens plane.

The effects of phase delays introduced by a smooth phase function can berelated to an equivalent, physical refractive lens under the paraxialapproximation, which can be derived using either geometric optics orfrom the Hyugens principle. The paraxial approximation holds when sinθ≈θ. For a projection system in which |θ|≤12°, (in this example the fullrange corresponds to redirecting light from one side of the image to theother) the error in the paraxial approximation is less than 1%. Thisfacilitates optimizing directly for the phase surface.

Using the simple paraxial approximation sin ø≈ø, which is valid forsmall deflection angles, it is possible to show that the geometricdisplacement in the image plane is proportional to the gradient of thephase function.

With the paraxial approximation sin ϕ≈ϕ, which is valid for smalldeflection angles, we obtain in 2D that

$\begin{matrix}{{u - x} = {{{f \cdot \sin}\; \varphi} \approx {f \cdot {\frac{\partial{p\left( {x,y} \right)}}{\partial x}.}}}} & (1)\end{matrix}$

In 3D this leads to the following equation for the mapping between apoint x on the lens plane and a corresponding point u on the imageplane:

u(x)=x+f·∇p(x).  (2)

Intensity Modulation

With the above mapping, we can derive the intensity change associatedwith this distortion. Let dx be a differential area on the lens plane,and let du=m(x)·dx be the differential area of the corresponding regionon the image plane, where m(⋅) is a spatially varying magnificationfactor. The intensity on the image plane is then given as

$\begin{matrix}{{{i\left( {u(x)} \right)} = {{\frac{dx}{du}i_{0}} = {\frac{1}{m(x)}i_{0}}}},} & (3)\end{matrix}$

where i₀ is the intensity of the collimated light incident on the lensplane. In the following we will assume i₀=1 for simplicity of notation.This corresponds to uniform illumination of the lens plane.

The magnification factor m(⋅) can be expressed in terms of thederivatives of the mapping between the lens and image planes (also seeFIG. 2):

$\begin{matrix}{{m(x)} = {{{\left( {\frac{\partial}{\partial x}{u(x)}} \right) \times \left( {\frac{\partial}{\partial y}{u(x)}} \right)} \approx {1 + {f\frac{\partial^{2}}{\partial x^{2}}{p(x)}} + {f\frac{\partial^{2}}{\partial y^{2}}{p(x)}}}} = {1 + {f \cdot {{\nabla^{2}{p(x)}}.}}}}} & (4)\end{matrix}$

This yields the following expression for the intensity distribution onthe image plane:

$\begin{matrix}{{i\left( {x + {f \cdot {\nabla{p(x)}}}} \right)} = {\frac{1}{1 + {f \cdot {\nabla^{2}{p(x)}}}}.}} & (5)\end{matrix}$

In other words, the magnification, m, and therefore the intensity i(u)on the image plane can be directly computed from the Laplacian of thescalar phase function on the lens plane.

Optimization Problem

While it is possible to directly turn the image formation mode fromEquation 5 into an optimization problem, we found that we can achievebetter convergence by first linearizing the equation with a first-orderTaylor approximation, which yields

i(x+f·∇p(x))≈1−f·∇ ² p(x),  (6)

where the left hand side can be interpreted as a warped imagei_(p)(x)=i(x+f·∇p(x)) where the target intensity i(u) in the image planehas been warped backwards onto the lens plane using the distortion u(x)produced by a given phase function p(x).

From this image formation model one can construct the followingoptimization problem for determining the phase function p(x) for a giventarget image i(u):

{circumflex over (p)}(x)=argmin_(p(x))∫_(x)(i _(p)(x)−1+f·∇ ² p(x))²dx  (7)

where i_(p) is a warped image i_(p)(x)=i(x+f·∇p(x)) where the targetintensity i(u) in the image plane has been warped backwards onto thelens plane using the distortion u(x) produced by a given phase functionp(x).

This optimization problem can be solved by iterating between updates tothe phase function and updates to the warped image, as illustrated bythe following example Algorithm 0:

Algorithm 0 Freeform lens optimization // Initialization i_(p) ⁰x = i(u)while not converged do // phase update p^(k)(x) = argmin_(p(x)) ∫_(x)(i_(p) ^((k−1))(x) − 1 + f · ∇²p(x))²dx // image warp i_(p) ^((k))(x) =i(x + f · ∇p^(k)(x)) end while

After a straightforward discretization of i(⋅) and p(⋅) into pixels, thephase update corresponds to solving a linear least squares problem witha discrete Laplace operator as the system matrix. We can solve thispositive semi-definite system using any one of a number of differentalgorithms, including Conjugate Gradient (CG), BICGSTAB and QuasiMinimal Residual (QMR). Such algorithms may be performed by a program.The image warp corresponds to a simple texture mapping operation, whichcan be implemented efficiently on a GPU (graphics processor unit).

The convergence behavior of this algorithm is shown in FIG. 4 whichshows algorithm stages for six iterations. The target image i getsprogressively distorted through backwards warping onto the lens planei˜k) as the phase function p (k) converges towards a solution. Thealgorithm uses the undistorted target image to optimize an initial phasefunction. Using this phase function, we update the target image on thelens plane by backward warping the image-plane target. This processincreasingly distorts the target image for the modulator plane as thephase function converges. Although the backward warping step implies anon-convex objective function, we empirically find that we achieveconvergence in only a small number of iterations (5-10). Overallprocessing time can be further accelerated by processing lower imageresolutions first and upsampling the result.

Solution in the Fourier Domain

Convergence speed of this algorithm can be further improved byunderstanding that the computational cost of the method is due primarilyto the solution of large-scale biharmonic problems. For example, aKrylov subspace method (QMR) may be employed however convergence istypically slow due to difficulties in finding an effectivepreconditioner and the scale of the systems. Algorithms useful forefficient solution of biharmonic systems are an ongoing topic ofresearch, including, for example, preconditioning approaches [Silvesterand Mihajlović 2004], multigrid methods [Zhao 2004] and operatorsplitting schemes [Tang and Christov 2006]. Scaling these to themillions of degrees of freedom required for imaging problems in realtime is extremely challenging.

An alternative approach based upon proximal operators can allow theproblem to be expressed in the Fourier domain and consequently solvedefficiently using highly parallelizable fast Fourier transformlibraries. This alternative approach permits solutions to be obtained inreal time or near real time using commodity low cost data processors.

Mirror padding the input image as described, for example, in [Ng et al.1999] causes the system arising from the discretization of ∇⁴ to haveperiodic boundary conditions with pure-Neumann boundary conditions atthe nominal image edge. This is illustrated in FIG. 3. This modificationallows the product ∇⁴p in the objective function, Equation 7, to beexpressed as a convolution via the Fourier convolution theorem, whichallows much faster Fourier-domain solver to be used.

For periodic boundary conditions, this problem can be solved veryefficiently in Fourier-space by using proximal operators. Proximalmethods from sparse optimization allow for regularization to be imposedwithout destroying the structure of the system.

For an arbitrary convex function, F(z), the proximal operator,prox_(γF), (defined in Equation 8) acts like a single step of a trustregion optimization in which a value of z is sought that reduces F butdoes not stray too far from the input argument q:

$\begin{matrix}{{{prox}_{\gamma \; F}(q)} = {{{argmin}_{2}{F(z)}} + {\frac{\gamma}{2}{Pz}} - {{qP}_{2}^{2}.}}} & (8)\end{matrix}$

For a least-squares objective F(z)=½∥Az−b∥₂ ², the resulting proximaloperator is shown in Equation 9.

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b)  (9)

Since proximal operators contain a strictly convex regularization term,the whole operator is a strictly convex function even if F is onlyweakly convex. This property of proximal operators helps in designingalgorithms with rapid convergence. A straightforward fixed-pointoptimization algorithm, the proximal-point method [Parikh and Boyd2013], exploits this to optimize strictly or weakly convex functions byrepeatedly evaluating the proximal operator of the objective, i.e.z^(k+1)=prox_(γF)(z^(k)), until convergence to a minimizer of F. Sincethe proximal regularization term can also be expressed as a Toeplitzmatrix (simply the identity matrix), it does not destroy the circulantstructure of the problem nor does it alter the solution by imposingunneeded regularization.

By denoting the forward and inverse Fourier transforms as F( ) & F⁻¹( )respectively, complex conjugation by * and performing multiplication anddivision point-wise, the proximal operator for Equation 9 can bere-expressed in the Fourier domain as Equation 10 for Toeplitz matricesA.

$\begin{matrix}{{{prox}_{\gamma \; F}(q)} = {F^{- 1}\left( \frac{{{F(b)}{F(A)}^{*}} + {\gamma \; {F(q)}}}{{\left( {1 + \alpha} \right){F(A)}^{2}} + \gamma} \right)}} & (10)\end{matrix}$

The constant α≥0 has been added to regularize the solver by favoringsolutions with low curvature. This corresponds to solving a modifiedform of Equation 7 that imposes a penalty of α/2∥∇²p(x)∥², as shown inEquation 11

{circumflex over (p)}(x)=argmin_(p(x))∫_(x)(i _(p)(x)−1+f·∇ ² p(x))²dx+∫ _(x)(∇² p(x))² dx,  (11)

The effect of the parameter a is to favor smoother solutions than canotherwise be found. This helps to prevent the method from producingundesirable caustics in an attempt to achieve very bright highlights atthe expense of image quality in darker regions. The effect of the aparameter is shown in FIG. 13 for simulations.

By defining A=f∇² and b=1−i_(p) ^(k)(x) and q=p^(k)(x), the problemdescribed above can be solved iteratively in Fourier space usingAlgorithm 1. This change allows each iteration of the non-linear solveto be computed using one forward/inverse Fourier transform, one imagewarping and some minor, component-wise operations. As shown, Equation 11is a non-linear variant of a common proximal algorithm, theproximal-point method, which is a fixed-point algorithm for minimizingan arbitrary convex F consisting of recursively calling prox_(γF) byevaluating: p^(k+1)←prox_(γF)(p^(k)).

Algorithm 1 Paraxial caustics in Fourier space // Initialize phasesurface as a constant value p⁰(x) ← 0 // Initialize iteration counterand constant parameters A ← f∇² k ← 0 while k < k_(max) do // Warptarget image by current solution i_(p) ^(k)(x) ← i(x + f∇p^(k)(x)) //initialize right hand side of least-squares problem b ← 1 − i_(p)^(k)(x) // Update the current solution by evaluating // the proximaloperator in Equation 10 p^(k+1) (x) = prox_(γF) (p^(k) (x)) // updateiteration index k ← k + 1 end while // RETURN computed mapping returnp^(k) ^(max) (x)

The re-formulation of the algorithm results in orders of magnitudespeedup to the algorithm when executed on a CPU using FFT based solversover the QMR solver described above. If the per-frame computation timesfor a QMR solver are 20 minutes or more the Fourier version in Algorithm1 may take approximately 0.6 seconds at the same resolution (256×128) ona Core i5 desktop computer, a speedup of approximately 2000 times. Theconversion to Fourier domain solves also results in operations that aremore easily implemented to run in parallel on one or more GPUs. We haveimplemented the algorithm the algorithm both in C++ and in CUDA usingCUFFT for the forward and inverse Fourier transforms [NVIDIA]. The CUDA& CUFFT version of the code yields nearly a 150 times speedup over thesingle-threaded CPU version when run on a GeForce 770 GPU, resulting inroughly a 300,000 fold speedup over the naive CPU version implementedusing QMR. The algorithm described herein is the first freeform lensingmethod of which the inventors are aware that is capable of operating inreal-time, see Table 1. This is in contrast to methods such as[Schwartzburg et al. 2014], which produce satisfactory results, but haveruntimes roughly five orders of magnitude higher than our GPU algorithm.This currently prevents their use in real-time capable projectionsystems.

TABLE 1 Runtimes for various resolution inputs with 10 iterations ofAlgorithm 1 Algorithm Resolution Runtime CPU 256 × 128 600 ms GPU 256 ×128  4 ms GPU 480 × 270  14 ms GPU 960 × 540  52 ms GPU 1920 × 1080 212ms

The algorithm is very well suited to hardware implementation on devicessuch as GPUs, FPGAs or ASICs due to its use of highly parallel FFTs andcomponent-wise operations. We run Algorithm 1 for a fixed number ofiterations (typically 10). Convergence to a solution is rapid, requiringwell fewer than 10 iterations; however for hardware implementations itis highly desirable to have computation times that are independent offrame content. The choice of smoothing factor α can be somewhat contentdependent.

Simulation Results

Using the equivalence between physical lenses and phase functions allowssolid lens models to be generated for testing via geometric opticssimulation (we use Blender+LuxRender). Although these models may notsatisfy the paraxial approximation, they serve well for quickqualitative comparisons since thickness effects tend to manifest aslow-spatial frequency distortions. Examples are shown in FIGS. 12 and 13which illustrate the effect of mirror padding and the choice of arespectively. It is important to note that these distortions do notaffect the prototype projector results since the prototype meets theconditions of the paraxial approximation well.

When higher physical accuracy is required, one can apply Huygens-Fresnelsimulation, which approximates the (complex) incident illumination as asuper-position of (complex) point sources. Simulation results are shownin FIG. 18 and are in good agreement with experimentally observedresults (see e.g. the caustics on Marilyn's nose in the ‘Simulated’ and‘Direct’ images) although the increased cost of simulation limitsresolution to below the levels needed to resolve diffraction effectsfrom discrete pixels. Speckle from the laser light source is similarlynot modeled.

Based on these results, we conclude that the phase modulation performslargely as expected, and the primary limitations in image quality arediffraction artifacts and speckle.

Static Refractive Lenses

The phase function p(x) can be used directly to drive a digital phasemodulation display (see below). However, if instead, we would like tocreate a refractive lens surface out of a transparent material, thenthis phase function may be converted to a geometric model for the lensshape.

We can model a lens shape that is flat on one side and has a freeformheight field h(x) on the other side (see FIG. 3). In the (x,z) plane,the deflection angle ϕ is related to the incident (✓_(i)) and theexitant (θ_(o)) angles at the height field as follows

$\begin{matrix}{{{\frac{\partial{p(x)}}{\partial x} \approx \varphi} = {\theta_{0} - \theta_{i}}},} & (12)\end{matrix}$

The analogous relationship holds in the (y,z) plane.

In addition, the lens material has a refractive index of n. UsingSnell's law, and again the paraxial approximation, we obtain

$\begin{matrix}{{\frac{1}{n} = {\frac{\sin \; \theta_{i}}{\sin \; \theta_{o}} \approx \frac{\theta_{i}}{\theta_{o}}}},} & (13)\end{matrix}$

Using Equations 12 and 13, as well as θ_(i)≈∂h(x)/∂x, we can derive thelens shape as

$\begin{matrix}{{{h(x)} = {h_{0} + {\frac{1}{n - 1}{p(x)}}}},} & (14)\end{matrix}$

where h₀ is a base thickness for the lens.

The height h(x) is a linear function of the phase. The refractive indexn shows up only as a scalar multiplier to the phase function p(⋅). Sincep itself is approximately linear in the focus distance f, we can seethat uniform scaling of the height field and uniform changes of therefractive index simply manifest themselves as a refocusing of the lens.This also shows that it is equivalently possible to adjust the exampleoptimization procedure proposed above to directly optimize for h(⋅)instead of p(⋅). The formulation above may be preferable in cases whereone is seeking to control only because a spatial phase modulator forexample for applications in video projectors.

FIG. 5 and the right-hand image of FIG. 5A show some example 3D printedrefractive lenses. In FIG. 5, the left image shows the lensesthemselves, while the center and right images show the causticsgenerated by them (the Lena image and a Siggraph logo). Due toresolution limits on the 3D printer, the lens dimensions have beenoptimized for large feature scales, which results in a short focallength.

FIG. 5 and the right-hand image of FIG. 5A show results for goal-basedcaustics using refractive freeform lenses generated with our method. Thelenses (shown on the left of FIG. 5) were 3D printed on an Objet Connex260 rapid prototyping machine using VeroClear™ material. Afterwards, thelenses were thoroughly cleaned and the flat side was manually polishedusing fine grained sand paper and polishing paste. This type of 3Dprinter has a layer thickness of 42 μm, which limits the feature sizethat can be readily created.

As discussed above, the model can be rescaled to achieve different focaldistances. To accommodate the resolution limits of the fabricationmethod, we chose very short focal distances f (about 1″ for the Siggraphlogo and 5″ for the Lena image). Although these scales test the verylimits of the paraxial approximation used in the derivation of our imageformation model, the image quality is still quite good. With betterfabrication methods such as injection molding, high precision milling oreven detailed manual polishing of a 3D printed surface, one could bothimprove the image quality and reduce the feature size, so that far fieldprojection becomes feasible.

Dynamic Lensing

In order to apply the freeform lens concept in projection displays, onemay apply a spatial light modulator that can manipulate the shape of thewavefront of reflected or transmitted light. Several differenttechnologies are available for this purpose.

Several adaptive optical devices lend themselves to the real-timevideo-capable implementation. Such devices includemicroelectromechanical systems (MEMS) based displays, such as the analog2D array of mirrors fabricated by [Hoskinson et al. 2012], or deformablemirrors used in wavefront sensing and correction applications.Continuous deformable mirrors [Menn et al. 2007] seem a particularlyattractive option since they eliminate diffraction due to regular pixelstructures. Although functioning mirrors with as many 4096 actuatorshave been reported, the spatial resolution of these MEMS-based devicesis still several order of magnitude lower than that of existing digitalmicro displays that are routinely used in digital projectors. This makestheir use at this point, less attractive in a dual-modulation setup.

Some embodiments advantageously apply wavefront modulators based onliquid crystal display (LCD) technology. LCDs are normally configured asamplitude (intensity) modulators by sandwiching them between two linearpolarization filters. However, when operated without the secondpolarizer, they retard (modulate) the phase of passing light differentlydepending on the rotation state of the liquid crystals in each pixel. Anelectric field across the cell gap of each pixel controls the amount ofphase retardation. In principle such a standard display is sufficient toimplement a dynamic lens. However there also exist dedicated,commercially available micro displays that have been optimized to a)maximize the amount of phase retardation (on the order of 2π and more)and to b) minimize the amount of polarization change. As such, the pixelvalues for this type of SLM correspond directly to our phase functionp(⋅) as derived above. A larger phase retardation allows for lenssurfaces with a steeper gradient, but comes at the cost of switchingspeed, as a thicker cell gap is required. If the phase change in the SLMdoes not affect polarization state (“phase-only”), this allows us to usethe display in combination with other opto-electronic components furtheralong the optical path, specifically a traditional amplitude SLM fordual modulation purposes. For further information on the topic we referto [Robinson et al. 2005].

An example prototype embodiment used a reflective Liquid Crystal onSilicon (LCoS) chip distributed by [HOLOEYE]. This chip has a spatialresolution of 1920×1080 discrete pixels at a pixel pitch of 6.4 μm, andcan be updated at up to 60 Hz. Access to a look-up-table allows forcalibration of the modulator for different working wavelengths. The fillfactor and reflectivity of the display are high compared to othertechnologies at 93% and 75% respectively. The phase retardation iscalibrated to between 0 and 2π, equivalent to one wavelength of light.This is sufficient to generate freeform lenses with a long focaldistance. For shorter focal distances, we require more strongly curvedwavefronts, which creates larger values for p(⋅). We can address thisissue by phase wrapping, i.e. just using the fractional part of p(⋅) todrive the SLM. This results in a pattern similar to a Fresnel lens.

We built two test beds. A first prototype contained a phase SLM withouta second amplitude modulator, and is reconfigurable between two types oflight source: a red 632.8 nm HeNe laser, and a white LED. This prototypeallows us to test the freeform lensing approach in isolation, and toevaluate artifacts such as diffraction based on light source type. Asecond prototype is a full dual-modulation projector using a green 532nm diode pumped solid state (DPSS) laser as a light source.

We first implemented a laser based system using a HeNe gas laser due toits good beam quality and low power which makes it safe in experiments(FIG. 6, top). This setup allows us to confirm and analyze diffractionpatterns that we expect to observe.

A significant advantage of our method, which is based on refractiveprinciples, over diffraction based projection approaches [Slinger et al.2005] are reduced requirements of the light source. Where diffractionpatterns utilized in 2D holographic projections systems ideally requirespatially and temporally coherent light for image formation, ourapproach enables light redirection using partially collimated broadbandlight. This is advantageous as even recent laser-based projectionsystems require broadening of the to reduce artifacts such as screenspeckle contrast as well as observer metamerism.

We demonstrate a prototype using a single, white broadband LED as alight source. In this example the LED had a short wavelength lightemitting die (blue) and a conversion phosphor (green-yellow). See FIG.6, bottom.

We also applied our new image formation approach on a laser based systemusing a 532 nm DPSS laser (FIG. 16). In contrast to the LED approach,the optical power of the laser light source (500 mW) is sufficient torelay and magnify the resulting light intensity profiles onto a largerprojection screen for evaluation.

As anticipated and later confirmed by wavefront simulations (FIG. 18,second row) the use of single frequency lasers causes artifactsincluding noticeable screen speckle contrast and diffraction “fringes”due to interference (FIG. 18, third row). As previously mentioned theseartifacts can be reduced below the noticeable visible threshold by usingfor example a set of lasers with different center wavelengths orbroadband light source such as LED and lamps [2015]. A similar image“smoothing” effect can be achieved by spatially or temporally averagingthe image using for example a diffuser or commercially availablecontinuous deformable mirrors that introduces slight angular diversityin a pseudo-random fashion at high speeds. This is particularly usefulwhen constrained to using a narrowband light source such as in our testsetup. For ease of implementation we choose to use a thin film diffuserplaced in the intermediate image plane following the phase SLM. Photosof the “cleaned-up” intensity profiles can be seen in (FIG. 8, fourthrow).

We also demonstrate a first prototype of a high brightness, high dynamicrange projection system, in which we form an image based on our dynamiclensing method and provide additional sharpness and contrast using atraditional LCoS-based amplitude modulating display.

At a high level, the light path of a traditional projection systemincludes a high intensity light source and some form of beam shaping,for example beam expansion, collimation and homogenization, colorseparation and recombining optics. At the heart of the projector, asmall SLM attenuates the amplitude of light per pixel. Our prototyperetained this architecture but replace the uniform illumination modulewith both a laser illumination and a phase SLM (FIG. 7). Our lensingsystem is inserted between the light source and the existing SLM, andforms an approximate light distribution on an intermediate image planecoinciding with the SLM plane.

The freeform lensing approach redistributes light from dark imageregions to bright ones, thus increasing both contrast and local peakbrightness, which is known to have a significant impact on visualrealism [Rempel et al. 2011].

We initially use a crude forward image formation model for the phase SLMto predict the illumination profile present at the second,amplitude-only modulator. Given the phase function from the freeformlensing algorithm, the light distribution on the image plane ispredicted using the simple model from Equations 2 and 4. The amount ofsmoothness introduced at the diffuser at the intermediate image planecan be approximated using a blur kernel and the modulation patternrequired for the amplitude modulator is then obtained to introduce anymissing spatial information as well as additional contrast where needed.We note that careful calibration and characterization of the entireoptical system is required to optimally drive the SLMs. No significantefforts beyond careful spatial registration of the two images(illumination profile caused by phase retardation and amplitudemodulation on the SLM) and calibration to linear increments in lightintensity were performed for this work.

Similar to the case of flat panel HDR displays [Seetzen et al. 2004], wecan use a forward image formation model for the phase SLM to predict the“backlight” illumination for second, amplitude-only modulator. Themodulation pattern for the amplitude modulator may be obtained bydividing the HDR target image by the “backlight” pattern.

FIG. 18 shows a selection of simulated and experimental results for ourmethod. The first row of FIG. 18 (“Phase Patterns”) shows the phasepatterns computed by Algorithm 4.1 as applied to the phase modulatorwith black corresponding to no phase retardation and white correspondingto a retardation of 2π. These patterns illustrate how phase patternswith maximum phase retardation larger than 2π can be wrapped to themaximum phase retardation of the modulator, resulting in a patternsimilar to a Fresnel lens.

The second row of FIG. 18 (‘Simulation’) shows simulations of the phasepattern using the Huygens-Fresnel principle. Unlike geometric opticssimulations such as path tracing, these simulations are able to capturemany of the diffraction artifacts. The third row (“Direct”) shows photosof our prototype using only phase modulation that exhibit diffractionartifacts as well as noise due to laser speckle. These artifacts can bealmost entirely removed by introducing the diffuser in the fourth row ofFIG. 18 (“Diffused”); the photos for this row used identical camerasettings to the “Direct” row.

Phase Pattern: the phase pattern as computed by Algorithm 1.Simulation: Huygens-Fresnel simulation of predicted image.Direct: photograph of actual image without diffuser showing diffractionartifacts.Diffuser: by adding the a thin-film diffuser, artifacts such asdiffraction fringes nearly completely mitigated.Standard: photo of standard, amplitude modulation only projection usinga single amplitude modulator shows elevated black levels and lowcontrast.Proposed (HDR): Using our lensing approach redistributes light from darkregions to bright regions, resulting in improved black levels andincreased highlight intensity. The last two rows appear slightlydistorted due to an off-angle position of the camera which becamenecessary because of a short throw projection and close screen as wellas baffles to block ambient light effectively to capture the black levelof the system.

In the fifth row of FIG. 18 (“Standard”), we show photographs of ourdual-modulation projector operating using only the amplitude modulator.This is achieved by providing a constant valued phase function todisable light redistribution. The results are typical of single stageprojectors, leaked light pollutes black levels and overall contrast islow due to an inefficient use of available power limiting highlightintensity.

Finally in the last row of FIG. 18 (“Proposed (HDR)”), we show photos ofour proposed phase+amplitude dual modulation approach. These photos werecaptured with identical camera settings to the “Standard” results (fifthrow), and show that our method not only recovers better black levels butalso, as expected, increases the brightness of highlights byredistributing light from dark regions of the image to lighter regions.This makes better use of available power, enabling high-dynamic rangeprojection with drastically reduced power consumption when compared todual amplitude modulation approaches.

FIG. 5A (left) shows the Lena image reproduced on the white lightversion of this setup. As expected, the broadband illumination averagesout most of the diffraction artifacts, resulting only in a relativelysmall spatial blur, very similar to the backlight blur in the originaldual modulation work by Seetzen et al. [2004]. This blur can becalibrated easily and can be compensated for in a dual modulation setup.

Results from our dual modulation setup are shown in FIGS. 9 and 10. FIG.9 shows just the effect of the freeform lensing approach, with theamplitude SLM set to a constant value. As in the HeNe laser setup, wecan identify a range of diffraction artifacts, although they are lesspronounced here due to the larger focal distance, and the reduced usageof phase wrapping. FIG. 10 shows a result of the actual dual modulationapproach. The second modulator stage has increase contrast and addedsignificant detail, but cannot get rid of some of the high-frequencyartifacts.

The following references provide background information and are herebyincorporated herein by reference.

-   BERRY, M. 2006. Oriental magic mirrors and the Laplacian image.    European journal of physics 27, 1, 109.-   BIMBER, O., AND IWAI, D. 2008. Superimposing dynamic range. ACM    Trans. Graph. 27, 5, 150.-   BLACKHAM, G., AND NEALE, A., 1998. Image display apparatus,    March 18. EP Patent App. EPI9,970,306,624.-   BUCKLEY, E. 2008. 70.2: Invited paper: holographic laser projection    technology. In Proc. SID, vol. 39, 1074-1079.-   DAMBERG, G., SEETZEN, H., WARD, G., HEIDRICH, W., AND    WHITEHEAD, L. 2007. 3.2: High dynamic range projection systems. In    Proc. SID, vol. 38, Wiley Online Library, 4-7.-   DAMBERG, G., SEETZEN, H., WARD, G., KANG, M., LONGHURST, P.,    HEIDRICH, W., AND WHITEHEAD, L. 2007. High dynamic range projector.    Siggraph Emerging Technologies.-   FINCKH, M., DAMMERTZ, H., AND LENSCH, H. P. 2010. Geometry    construction from caustic images. In Proc. ECCV, 464-477.-   HAUGEN, P. R., BARTELT, H., AND CASE, S. K. 1983. Image formation by    multifacet holograms. Applied optics 22, 18, 2822-2829.-   HOLOEYE. Photonics corporation. URL http://www.holoeye.com.-   HOSKINSON, R., STOEBER, B., HEIDRICH, W., AND FELS, S. 2010. Light    reallocation for high contrast projection using an analog    micromirror array. ACM Transactions on Graphics (TOG) 29, 6, 165.-   HOSKINSON, R., HAMPL, S., AND STOEBER, B. 2012. Arrays of    large-area, tip/tilt micromirrors for use in a high-contrast    projector. Sensors and Actuators A: Physical 173, 1, 172-179.-   HULLIN, M. B., IHRKE, I., HEIDRICH, W., WEYRICH, T., DAMBERG, G.,    AND FUCHS, M. 2013. State of the art in computational fabrication    and display of material appearance. In Eurographies Annual    Conference (STAR).-   KISER, T., EIGENSATZ, M., NGUYEN, M. M., BOMPAS, P., AND    PAULY, M. 2013. Architectural caustics controlling light with    geometry. In Advances in Architectural Geometry 2012. Springer,    91-106.-   [Kusakabe et al. 2009] Kusakabe, Y., Kanazawa, M., Nojiri, Y.,    Furuya, M., and Yoshimura, M. 2009. A high-dynamic-range and    high-resolution projector with dual modulation. vol. 7241,    72410Q-72410Q-11.-   LESEM, L., HIRSCH, P., AND JORDAN, J. 1969. The kinoform: a new    wavefront reconstruction device. IBM Journal of Research and    Development 13, 2, 150-155.-   MENN, S., CORNELISSEN, S. A., AND BIERDEN, P. A. 2007. Advances in    mems deformable mirror technology for laser beam shaping. In    Photonic Devices+Applications, International Society for Optics and    Photonics, 66630M-66630M.-   MINANO, J. C., BENITEZ, P., AND SANTAMARIA, A. 2009. Freeform optics    for illumination. Optical Review 16, 2, 99-102.-   [Ng et al. 1999] Ng, M. K., Chan, R. H., and Tang, W.-C. 1999. A    fast algorithm for deblurring models with neumann boundary    conditions. SIAM Journal on Scientific Computing 21, 3, 851-866.-   [NVIDIA] NVIDIA, C. Programming guide, cusparse, cublas, and cufft    library user guides. {Online}.-   PAPAS, M., JAROSZ, W., JAKOB, W., RUSINKIEWICZ, S., MATUSIK,W., AND    WEYRICH, T. 2011. Goal-based caustics.Computer Graphics Forum 30, 2,    503-511.-   PAPAS, M., HOUIT, T., NOWROUZEZAHRAI, D., GROSS, M., AND    JAROSZ, W. 2012. The magic lens: refractive steganography. ACM    Transactions on Graphics (TOG) 31, 6, 186.-   [Parikh and Boyd 2013] Parikh, N., and Boyd, S. 2013. Proximal    algorithms. Foundations and Trends in Optimization 1, 3, 123-231.-   PAULY, M., AND KISER, T. 2012. Caustic art. Tech. rep.-   REMPEL, A., HEIDRICH, W., LI, H., AND MANTIUK, R. 2009. Video    viewing preferences for hdr displays under varying ambient    illumination. Proc. APGV, 45-52.-   REMPEL, A., HEIDRICH, W., AND MANTIUK, R. 2011. The role of contrast    in the perceived depth of monocular imagery. Proc. APGV, 115.-   ROBINSON, M. D., SHARP, G., AND CHEN, J. 2005. Polarization    engineering for LCD projection, vol. 4. John Wiley & Sons.-   SCHWARTZBURG, Y., TESTUZ, R., TAGLIASACCHI, A., AND PAULY, M. 2014.    High-contrast computational caustic design. ACM Trans. Graph. (Proc.    Siggraph). (in print).-   SEETZEN, H., HEIDRICH, W., STUERZLINGER, W., WARD, G., WHITEHEAD,    L., TRENTACOSTE, M., GHOSH, AND VOROZCOVS, A. 2004. High dynamic    range display systems. ACM Trans. Graph. (Proc. SIGGRAPH) (August),    760-768.-   SEETZEN, H. 2009. High dynamic range display and projection systems.    PhD thesis, University of British Columbia.-   [Silvester and MihajloviÀ 2004] Silvester, D. J., and    Mihajlović, M. D. 2004. A black-box multigrid preconditioner for the    biharmonic equation. BIT Numerical Mathematics 44, 1, 151-163.-   SLINGER, C., CAMERON, C., AND STANLEY, M. 2005. Computer-generated    holography as a generic display technology. IEEE Computer 38, 8,    46-53.-   [Tang and Christov 2006] Tang, X. H., and Christov, C. I. 2006. An    operator splitting scheme for biharmonic equation with accelerated    convergence. In Proceedings of the 5th International Conference on    Large-Scale Scientific Computing, Springer-Verlag, Berlin,    Heidelberg, LSSC'05, 387-394.-   YUE, Y., IWASAKI, K., CHEN, B.-Y., DOBASHI, Y., AND    NISHITA, T. 2012. Pixel art with refracted light by rearrangeable    sticks. Computer Graphics Forum 31, 2pt3, 575-582.-   YUE, Y., IWASAKI, K., CHEN, B.-Y., DOBASHI, Y., AND    NISHITA, T. 2014. Poisson-based continuous surface generation for    goal-based caustics. ACM Trans. Graph. (in print).-   [Zhao 2004] Zhao, J. 2004. Convergence of v-cycle and f-cycle    multigrid methods for the biharmonic problem using the morley    element. Electronic Transactions on Numerical Analysis 17, 112-132.

It can be appreciated that some embodiments provide one or more of thefollowing:

-   -   A new, algorithm for freeform lens optimization (“goal-based        caustics”) that is dramatically simpler than some prior art        algorithms. The algorithm may be applied to controlling the        projection of light in real time or near real time.    -   Some embodiments operate directly in phase space and therefore        can be implemented as iterative methods that can not only        generate modulation patterns for a phase modulator, but also for        conventional refractive lenses without additional steps such as        Poisson integration.    -   A new dual-modulation projector design that combines one phase        and one amplitude modulator for image generation and is capable        of working white (incoherent) light. To our knowledge,    -   Methods and apparatus as described herein may also be applied        for generating static light fields useful, for example, for        architectural lighting and/or vehicle lighting.    -   direct optimization for the modulated phase of the light, no        need to trade off between data term and integrability of the        surface    -   made possible by finding a parameterization of the problem that        allows us to express the optimization in the modulator/lens        plane rather than the image plane.    -   our derivation relies on small angle image formation (paraxial        approximation), which is well established in the optics        community.

Interpretation of Terms

Unless the context clearly requires otherwise, throughout thedescription and the claims:

-   -   “comprise”, “comprising”, and the like are to be construed in an        inclusive sense, as opposed to an exclusive or exhaustive sense;        that is to say, in the sense of “including, but not limited to”;    -   “connected”, “coupled”, or any variant thereof, means any        connection or coupling, either direct or indirect, between two        or more elements; the coupling or connection between the        elements can be physical, logical, or a combination thereof;    -   “herein”, “above”, “below”, and words of similar import, when        used to describe this specification, shall refer to this        specification as a whole, and not to any particular portions of        this specification;    -   “or”, in reference to a list of two or more items, covers all of        the following interpretations of the word: any of the items in        the list, all of the items in the list, and any combination of        the items in the list;    -   the singular forms “a”, “an”, and “the” also include the meaning        of any appropriate plural forms.

Words that indicate directions such as “vertical”, “transverse”,“horizontal”, “upward”, “downward”, “forward”, “backward”, “inward”,“outward”, “vertical”, “transverse”, “left”, “right”, “front”, “back”,“top”, “bottom”, “below”, “above”, “under”, and the like, used in thisdescription and any accompanying claims (where present), depend on thespecific orientation of the apparatus described and illustrated. Thesubject matter described herein may assume various alternativeorientations. Accordingly, these directional terms are not strictlydefined and should not be interpreted narrowly.

Embodiments of the invention may be implemented using specificallydesigned hardware, configurable hardware, programmable data processorsconfigured by the provision of software (which may optionally comprise“firmware”) capable of executing on the data processors, special purposecomputers or data processors that are specifically programmed,configured, or constructed to perform one or more steps in a method asexplained in detail herein and/or combinations of two or more of these.Examples of specifically designed hardware are: logic circuits,application-specific integrated circuits (“ASICs”), large scaleintegrated circuits (“LSIs”), very large scale integrated circuits(“VLSIs”), and the like. Examples of configurable hardware are: one ormore programmable logic devices such as programmable array logic(“PALs”), programmable logic arrays (“PLAs”), and field programmablegate arrays (“FPGAs”)). Examples of programmable data processors are:microprocessors, digital signal processors (“DSPs”), embeddedprocessors, graphics processors, math co-processors, general purposecomputers, server computers, cloud computers, mainframe computers,computer workstations, and the like. For example, one or more dataprocessors in a control circuit for a device may implement methods asdescribed herein by executing software instructions in a program memoryaccessible to the processors.

Processing may be centralized or distributed. Where processing isdistributed, information including software and/or data may be keptcentrally or distributed. Such information may be exchanged betweendifferent functional units by way of a communications network, such as aLocal Area Network (LAN), Wide Area Network (WAN), or the Internet,wired or wireless data links, electromagnetic signals, or other datacommunication channel.

For example, while processes or blocks are presented in a given order,alternative examples may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times.

In addition, while elements are at times shown as being performedsequentially, they may instead be performed simultaneously or indifferent sequences. It is therefore intended that the following claimsare interpreted to include all such variations as are within theirintended scope.

Software and other modules may reside on servers, workstations, personalcomputers, tablet computers, image data encoders, image data decoders,PDAs, color-grading tools, video projectors, audio-visual receivers,displays (such as televisions), digital cinema projectors, mediaplayers, and other devices suitable for the purposes described herein.Those skilled in the relevant art will appreciate that aspects of thesystem can be practised with other communications, data processing, orcomputer system configurations, including: Internet appliances,hand-held devices (including personal digital assistants (PDAs)),wearable computers, all manner of cellular or mobile phones,multi-processor systems, microprocessor-based or programmable consumerelectronics (e.g., video projectors, audio-visual receivers, displays,such as televisions, and the like), set-top boxes, network PCs,mini-computers, mainframe computers, and the like.

The invention may also be provided in the form of a program product. Theprogram product may comprise any non-transitory medium which carries aset of computer-readable instructions which, when executed by a dataprocessor, cause the data processor to execute a method of theinvention. Program products according to the invention may be in any ofa wide variety of forms. The program product may comprise, for example,non-transitory media such as magnetic data storage media includingfloppy diskettes, hard disk drives, optical data storage media includingCD ROMs, DVDs, electronic data storage media including ROMs, flash RAM,EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductorchips), nanotechnology memory, or the like. The computer-readablesignals on the program product may optionally be compressed orencrypted.

In some embodiments, the invention may be implemented in software. Forgreater clarity, “software” includes any instructions executed on aprocessor, and may include (but is not limited to) firmware, residentsoftware, microcode, and the like. Both processing hardware and softwaremay be centralized or distributed (or a combination thereof), in wholeor in part, as known to those skilled in the art. For example, softwareand other modules may be accessible via local memory, via a network, viaa browser or other application in a distributed computing context, orvia other means suitable for the purposes described above. In someembodiments image data is processed by a processor executing softwareinstructions to yield control signals for a phase modulator. Thesoftware may execute in real time in some embodiments (other embodimentsare also possible).

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

Specific examples of systems, methods and apparatus have been describedherein for purposes of illustration. These are only examples. Thetechnology provided herein can be applied to systems other than theexample systems described above. Many alterations, modifications,additions, omissions, and permutations are possible within the practiceof this invention. This invention includes variations on describedembodiments that would be apparent to the skilled addressee, includingvariations obtained by: replacing features, elements and/or acts withequivalent features, elements and/or acts; mixing and matching offeatures, elements and/or acts from different embodiments; combiningfeatures, elements and/or acts from embodiments as described herein withfeatures, elements and/or acts of other technology; and/or omittingcombining features, elements and/or acts from described embodiments.

The following are non-limiting enumerated example embodiments of theinvention:

-   1. A method for controlling a phase modulator to display a target    light pattern defined by image data, the method comprising:    -   initializing a warped image based on the image data, the warped        image warped from the target light pattern by distortions        corresponding to a phase function p(x) representing a phase        shift applied by the phase modulator for regions in the lens        plane;    -   refining the phase function and the warped image by performing a        plurality of iterations wherein each of the plurality of        iterations includes:        -   a step of updating the phase function by performing an            optimization which yields an updated phase function wherein            the updated phase function reduces a difference measure            between the warped image and the inverse of a magnification            provided by the phase function at points in the warped            image; and        -   a step of warping the target light pattern onto the lens            plane using a distortion u(x) produced by the updated phase            function p(x) to yield an updated warped image.-   2. A method according to enumerated example embodiment 1 wherein the    difference measure comprises a sum of squares of differences between    pixels of the warped image and inverses of the magnification at the    points in the warped image.-   3. A method according to enumerated example embodiment 1 or 2    wherein the updating the phase function comprises computing    differences between pixels of the warped image and corresponding    values for 1−f·∇²p(x).-   4. A method according to enumerated example embodiment 1 or 2    wherein the updating the phase function comprises computing    differences between pixels of the warped image and corresponding    values for 1/(1+f·∇²p(x)).-   5. A method according to any one of enumerated example embodiments 1    to 3 wherein the step of updating the phase function comprises    solving a linear least squares problem.-   6. A method according to enumerated example embodiment 4 wherein the    least squares problem comprises a system matrix comprising a    discrete Laplace operator.-   7. A method according to enumerated example embodiment 1 wherein the    step of updating the phase function comprises solving:

{circumflex over (p)}(x)=argmin_(p(x))∫_(x)(i _(p)(x)−1+f·∇ ² p(x))² dx

-   8. A method according to any one of enumerated example embodiments 4    to 6 wherein performing the optimization comprises applying an    algorithm selected from the group consisting of: Conjugate Gradient    (CG), BICGSTAB and Quasi Minimal Residual (QMR).-   9. A method according to any one of enumerated example embodiments 1    to 7 wherein the step of warping the target intensity in the image    plane backwards onto the lens plane comprises performing a texture    mapping operation.-   10. A method according to enumerated example embodiment 8 wherein    the texture mapping operation is implemented on a graphics processor    unit.-   11. A method according to enumerated example embodiment 8 or 9    wherein the step of warping the target intensity in the image plane    backwards onto the lens plane comprises computing:

i _(p)(x)=i(x+f·∇p(x))

-   12. A method according to any one of enumerated example embodiments    1 to 11 comprising modelling blur in an image at the image plane and    generating control values for an amplitude modulator that tend to    compensate at least in part for the blur.-   13. A method according to any one of enumerated example embodiments    1 to 12 comprising displaying the target light pattern by    controlling the phase modulator according to the phase function and    illuminating the phase modulator with light.-   14. A method according to enumerated example embodiment 13 wherein    the light is broadband light.-   15. A method according to enumerated example embodiment 14 wherein    the broadband light is white light.-   16. A method according to enumerated example embodiment 13 wherein    the light is monochromatic.-   17. A method according to enumerated example embodiment 13 or 16    wherein the light is laser light.-   18. A method according to any one of enumerated example embodiments    13 to 17 wherein the light is collimated.-   19. A method according to enumerated example embodiment 18 wherein    the light is incident on the phase modulator in a direction normal    to the lens plane.-   20. A method according to any one of enumerated example embodiments    1 to 19 wherein the light pattern comprises one or more bright spots    of light.-   21. A method according to enumerated example embodiment 20    comprising controlling the phase function applied to the phase    modulator to cause the one or more bright spots of light to move.-   22. A method according to enumerated example embodiment 20 or 21    wherein the one or more bright spots of light have intensities    exceeding a maximum uniform illumination intensity at the image    plane.-   23. A method according to any one of enumerated example embodiments    1 to 22 wherein a resolution of the phase modulator is at least 1    Megapixels.-   24. A method according to enumerated example embodiment 23 wherein    the phase modulator comprises at least 5 Megapixels.-   25. A method according to any one of enumerated example embodiments    1 to 24 wherein the light pattern occupies an image area in the    image plane and, from any point on the phase modulator, a light ray    directed from the point to any point on a boundary of the image area    forms an angle θ to a normal of the phase modulator wherein |θ|≤12°.-   26. A method according to any one of enumerated example embodiments    1 to 24 wherein a numerical aperture for points in the lens plane is    such that the paraxial approximation holds to within 1%.-   27. A method according to any one of enumerated example embodiments    1 to 26 wherein initializing the warped image comprises setting the    warped image to be the same as the target light pattern.-   28. A method according to any one of enumerated example embodiments    1 to 27 wherein the phase modulator comprises a liquid crystal phase    modulator.-   29. A method according to enumerated example embodiment 28 wherein    the phase modulator comprises a LCoS device.-   30. A method according to any one of enumerated example embodiments    1 to 27 wherein the phase modulator comprises a variable mirror.-   31. A method according to any one of enumerated example embodiments    1 to 30 wherein the image data comprises video data having a frame    rate of at least 20 frames per second.-   32. A method according to enumerated example embodiment 31 wherein    the video data provides a different target light pattern for each    frame and the method comprises calculating a different phase    function for each frame.-   33. A method according to enumerated example embodiment 32    comprising calculating the different phase functions in real time.-   34. A method according to any one of enumerated example embodiments    1 to 33 wherein refining the phase function and the warped image is    performed in 10 or fewer of the iterations.-   35. A method according to any one of enumerated example embodiments    1 to 34 wherein refining the phase function and the warped image is    performed in a fixed number of the iterations.-   36. A method according to any one of enumerated example embodiments    1 to 35 comprising executing one or more steps of refining the phase    function and the warped image in parallel in one or more graphics    processor units.-   37. A method according to any one of enumerated example embodiments    1 to 35 comprising performing at least some steps of refining the    phase function and the warped image in a frequency domain.-   38. A method according to enumerated example embodiment 37    comprising generating an optimization function; performing a Fourier    transform on the warped image, generating the phase function in the    frequency domain using the Fourier transform of the warped image and    performing an inverse Fourier transformation on the phase function.-   39. A method according to enumerated example embodiment 38    comprising performing the Fourier transform in hardware configured    to perform the Fourier transform.-   40. A method according to any one of enumerated example embodiments    37 to 39 comprising, before performing the steps in the frequency    domain, extending the image data to have periodic boundary    conditions.-   41. A method according to enumerated example embodiment 40 wherein    extending the image data comprises making a mirror image of the    image data across each boundary of the image data.-   42. A method according to any one of enumerated example embodiments    1 to 41 comprising generating control signals for a spatial light    modulator to correct intensities of light modulated by the phase    modulator.-   43. A method according to any one of enumerated example embodiments    1 to 42 comprising performing one or more of the iterations at a    first spatial resolution and upsampling the updated phase function    yielded by the one or more of the iterations.-   44. A method according to enumerated example embodiment 43    comprising, subsequent to upsampling the updated phase function,    performing one or more additional ones of the iterations at a second    resolution higher than the first resolution.-   45. A method according to any one of enumerated example embodiments    1 to 44 wherein the image data comprises video data, the target    light pattern is defined for one of the frames of the image data and    different target light patterns are defined in the image data for    other frames of the image data.-   46. Apparatus for controlling a phase modulator to display a target    light pattern defined by image data, the apparatus comprising a data    processor in communication with the phase modulator, the data    processor configured to:    -   receive the image data as input;    -   initialize a warped image based on the image data, the warped        image warped from the target light pattern by distortions        corresponding to a phase function p(x) representing a phase        shift applied by the phase modulator for regions in the lens        plane;    -   refine the phase function and the warped image by performing a        plurality of iterations wherein each of the plurality of        iterations includes:        -   a step of updating the phase function by performing an            optimization which yields an updated phase function wherein            the updated phase function reduces a difference measure            between the warped image and the inverse of a magnification            provided by the phase function at points in the warped            image; and        -   a step of warping the target light pattern onto the lens            plane using a distortion u(x) produced by the updated phase            function p(x) to yield an updated warped image; and    -   generate control signals for the phase modulator based on the        refined phase function.-   47. Apparatus according to enumerated example embodiment 46 wherein    the difference measure comprises a sum of squares of differences    between pixels of the warped image and inverses of the magnification    at the points in the warped image.-   48. Apparatus according to enumerated example embodiment 46 or 47    wherein the step of updating the phase function comprises computing,    by the data processor, differences between pixels of the warped    image and corresponding values for −1+f·∇²p(x).-   49. Apparatus according to enumerated example embodiment 46 or 47    wherein the step of updating the phase function comprises computing,    by the data processor, differences between pixels of the warped    image and corresponding values for 1/(1+f·∇²p(x)).-   50. Apparatus according to any one of enumerated example embodiments    46 to 48 wherein the step of updating the phase function comprises    solving, by the data processor, a linear least squares problem.-   51. Apparatus according to enumerated example embodiment 49 wherein    the least squares problem comprises a system matrix comprising a    discrete Laplace operator.-   52. Apparatus according to enumerated example embodiment 46 wherein    the step of updating the phase function comprises solving, by the    data processor:

{circumflex over (p)}(x)=argmin_(p(x))∫_(x)(i _(p)(x)−1+f·∇ ² p(x))² dx

-   53. Apparatus according to any one of enumerated example embodiments    49 to 51 wherein performing the optimization comprises applying, by    the data processor, an algorithm selected from the group consisting    of: Conjugate Gradient (CG), BICGSTAB and Quasi Minimal Residual    (QMR).-   54. Apparatus according to any one of enumerated example embodiments    46 to 52 wherein the step of warping the target intensity in the    image plane backwards onto the lens plane comprises performing a    texture mapping operation.-   55. Apparatus according to enumerated example embodiment 53    comprising a graphics processor unit and wherein the texture mapping    operation is implemented on the graphics processor unit.-   56. Apparatus according to enumerated example embodiment 53 or 54    wherein the step of warping the target intensity in the image plane    backwards onto the lens plane comprises computing, by the data    processor:

i _(p)(x)=i(x+f·∇′p(x))

-   57. Apparatus according to any one of enumerated example embodiments    46 to 56 wherein the data processor is configured to model blur in    an image at the image plane and generate control values for an    amplitude modulator that tend to compensate at least in part for the    blur.-   58. Apparatus according to any one of enumerated example embodiments    46 to 57 comprising the phase modulator and a light source for    projecting light on the phase modulator, wherein the data processor    is configured to generate the target light pattern by controlling    the phase modulator according to the phase function and controlling    the light source to illuminate the phase modulator with light.-   59. Apparatus according to enumerated example embodiment 58 wherein    the light is broadband light.-   60. Apparatus according to enumerated example embodiment 59 wherein    the broadband light is white light.-   61. Apparatus according to enumerated example embodiment 58 wherein    the light is monochromatic.-   62. Apparatus according to enumerated example embodiment 59 or 61    wherein the light is laser light.-   63. Apparatus according to any one of enumerated example embodiments    58 to 62 wherein the light is collimated.-   64. Apparatus according to enumerated example embodiment 63 wherein    the light source is configured to project light incident on the    phase modulator in a direction normal to the lens plane.-   65. Apparatus according to any one of enumerated example embodiments    58 to 64 wherein a resolution of the phase modulator is at least 1    Megapixels.-   66. Apparatus according to enumerated example embodiment 65 wherein    the resolution of the phase modulator is at least 5 Megapixels.-   67. Apparatus according to any one of enumerated example embodiments    58 to 66 wherein the target light pattern occupies an image area in    the image plane and, from any point on the phase modulator, a light    ray directed from the point to any point on a boundary of the image    area forms an angle θ to a normal of the phase modulator wherein    |θ|≤12°.-   68. Apparatus according to any one of enumerated example embodiments    58 to 67 wherein the phase modulator comprises a liquid crystal    phase modulator.-   69. Apparatus according to enumerated example embodiment 68 wherein    the phase modulator comprises a LCoS device.-   70. Apparatus according to any one of enumerated example embodiments    58 to 67 wherein the phase modulator comprises a variable mirror.-   71. Apparatus according to any one of enumerated example embodiments    46 to 70 wherein the target light pattern comprises one or more    bright spots of light.-   72. Apparatus according to enumerated example embodiment 71 wherein    the data processor is configured to control the phase function    applied to the phase modulator to cause the one or more bright spots    of light to move.-   73. Apparatus according to enumerated example embodiment 71 or 72    wherein the one or more bright spots of light have intensities    exceeding a maximum uniform illumination intensity at the image    plane.-   74. Apparatus according to any one of enumerated example embodiments    46 to 73 wherein a numerical aperture for points in the lens plane    is such that the paraxial approximation holds to within 1%.-   75. Apparatus according to any one of enumerated example embodiments    46 to 74 wherein the data processor being configured to initialize    the warped image comprises the data processor being configured to    set the warped image to be the same as the target light pattern.-   76. Apparatus according to any one of enumerated example embodiments    46 to 75 wherein the image data comprises video data having a frame    rate of at least 20 frames per second.-   77. Apparatus according to enumerated example embodiment 76 wherein    the video data provides a different target light pattern for each    frame and the data processor is configured to calculate a different    phase function for each frame.-   78. Apparatus according to enumerated example embodiment 77 wherein    the data processor is configured to calculate the different phase    functions in real time.-   79. Apparatus according to any one of enumerated example embodiments    46 to 78 wherein the data processor is configured to refine the    phase function and the warped image in 10 or fewer iterations.-   80. Apparatus according to any one of enumerated example embodiments    46 to 78 wherein the data processor is configured to refine the    phase function and the warped image in a fixed number of the    iterations.-   81. Apparatus according to any one of enumerated example embodiments    46 to 78 comprising one or more graphics processor units wherein the    data processor is configured to execute one or more steps of    refining the phase function and the warped image in parallel in the    one or more graphics processor units.-   82. Apparatus according to any one of enumerated example embodiments    46 to 80 wherein the data processor is configured to perform at    least some steps of refining the phase function and the warped image    in a frequency domain.-   83. Apparatus according to enumerated example embodiment 82 wherein    the data processor is configured to: perform a Fourier transform on    the warped image, generate the phase function in the frequency    domain using the Fourier transform of the warped image and perform    an inverse Fourier transformation on the phase function.-   84. Apparatus according to enumerated example embodiment 83    comprising a hardware Fourier transform device wherein the data    processor is configured to control the Fourier transform device to    perform the Fourier transform.-   85. Apparatus according to any one of enumerated example embodiments    46 to 84 comprising a spatial light modulator wherein the data    processor is configured to apply control signals to the spatial    light modulator to correct intensities of light modulated by the    phase modulator.-   86. Apparatus according to any one of enumerated example embodiments    82 to 84 wherein the data processor is configured to extend the    image data to have periodic boundary conditions before performing    the steps in the frequency domain.-   87. Apparatus according to enumerated example embodiment 86 wherein    extending the image data comprises making a mirror image of the    image data across each boundary of the image data.-   88. Apparatus according any one of enumerated example embodiments 46    to 87 comprising generating control signals for a spatial light    modulator to correct intensities of light modulated by the phase    modulator.-   89. Apparatus according to any one of enumerated example embodiments    46 to 88 wherein the data processor is configured to perform one or    more of the iterations at a first spatial resolution and upsample    the updated phase function yielded by the one or more of the    iterations.-   90. Apparatus according to enumerated example embodiment 89 wherein    the data processor is configured to perform one or more additional    ones of the iterations at a second resolution higher than the first    resolution, subsequent to upsampling the updated phase function.-   91. Apparatus according any one of enumerated example embodiments 46    to 90 wherein the image data comprises video data, the target light    pattern is defined for one of the frames of the image data and    different target light patterns are defined in the image data for    other frames of the image data.-   92. A method for generating control values for a phase modulator    from image data defining a target light pattern, the method    comprising:    -   establishing a mapping between points in the light pattern and        corresponding points on the phase modulator;    -   using the mapping, deriving a phase function, p, that includes        the control values by mapping the target light pattern into a        coordinate space of the phase modulator; and    -   processing the mapped target light pattern in the coordinate        space of the phase modulator.-   93. The method according to enumerated example embodiment 92 wherein    processing the mapped target light pattern comprises optimizing a    trial phase function based on comparisons of intensities in the    mapped target light pattern at the points on the phase modulator to    corresponding optical properties of the phase function in    neighborhoods of the points.-   94. The method according to enumerated example embodiment 93 wherein    the corresponding optical properties comprise magnifications.-   95. The method according to any one of enumerated example    embodiments 93 and 94 comprising determining the optical properties    based on a Laplacian of the phase function at the corresponding    points.-   96. The method according to enumerated example embodiment 95    comprising determining the Laplacian of the phase function using a    discrete Laplacian operator.-   97. A method for displaying video data, the video data specifying    video frames for display at a frame rate, the method comprising:    -   in real time, processing the video data to yield a sequence of        phase-modulator control signals at the frame rate,    -   applying the phase modulator control signals to an illuminated        two-dimensional spatial phase modulator, and    -   directing resulting phase-modulated light to a viewing area.-   98. The method according to enumerated example embodiment 97    comprising further amplitude-modulating the phase-modulated light.-   99. The method according to enumerated example embodiment 98 wherein    further amplitude-modulating the phase-modulated light comprises    controlling a spatial light modulator in a path of the    phase-modulated light.-   100. The method according to enumerated example embodiment 99    comprising computing a blur in the phase-modulated light and    controlling the spatial light modulator to reduce the blur.-   101. The method according to any one of enumerated example    embodiments 97 to 100 wherein processing the video data comprises:    -   establishing a mapping between points in the light pattern and        corresponding points on the light modulator;    -   using the mapping, deriving a phase function, p, that includes        the control values by mapping the target light pattern into a        coordinate space of the phase modulator; and    -   processing the mapped target light pattern in the coordinate        space of the phase modulator.-   102. The method according to enumerated example embodiment 101    wherein processing the mapped target light pattern comprises    optimizing a trial phase function based on comparisons of    intensities in the mapped target light pattern at the points on the    phase modulator to corresponding optical properties of the phase    function in neighborhoods of the points.-   103. The method according to enumerated example embodiment 102    wherein the corresponding optical properties comprise    magnifications.-   104. The method according to enumerated example embodiment 102 or    103 comprising determining the optical properties based on a    Laplacian of the phase function at the corresponding points.-   105. The method according to enumerated example embodiment 104    comprising determining the Laplacian of the phase function using a    discrete Laplacian operator.-   106. The method according to any one of enumerated example    embodiments 97 to 102 wherein processing the video data is performed    in a frequency domain.-   107. The method according to enumerated example embodiment 106    wherein processing the video data comprises generating an    optimization function; performing a Fourier transform on the    optimization function generating the phase function in the frequency    domain and performing an inverse Fourier transformation on the phase    function.-   108. The method according to enumerated example embodiment 107    comprising performing the Fourier transform in hardware configured    to perform the Fourier transform.-   109. The method according to any one of enumerated example    embodiments 92 to 96 and 101 to 108 wherein the phase modulator has    a maximum phase retardation and the method comprises subtracting a    multiple of 2π from phase shifts of the phase function that exceed    the maximum phase retardation of the phase modulator.-   110. A method for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   determining an objective function based on the image data;    -   transforming the objective function into a frequency space;    -   minimizing the transformed objective function in the frequency        space to obtain a phase function in the frequency space; and    -   inverse transforming the phase function to obtain a solution        phase function relating the phase of the phase modulator to a        position in two dimensions.-   111. The method according to enumerated example embodiment 110    wherein transforming the objective function comprises computing a    Fourier transform of the objective function.-   112. The method according to enumerated example embodiment 111    comprising, before transforming, extending the image data to have    periodic boundary conditions and basing the objective function on    the extended image data.-   113. The method according to enumerated example embodiment 112    wherein extending the image data comprises making a mirror image of    the image data across each boundary of the image data.-   114. The method according to any one of enumerated example    embodiments 110 to 113 wherein the objective function is a least    squares objective function.-   115. The method according to any one of enumerated example    embodiments 110 to 114 wherein the objective function includes a    cost for deviating from an input argument.-   116. The method according to any one of enumerated example    embodiments 110 to 115 wherein the method is performed iteratively    and in each of a plurality of iterations the input argument for the    objective function is the solution phase function for a previous    iteration.-   117. The method according to enumerated example embodiment 116    comprising caching a Fourier transform of the solution phase    function for the previous iteration and applying the cached Fourier    transform of the solution phase function in a current iteration.-   118. The method according to any one of enumerated example    embodiments 110 to 117 wherein the objective function comprises a    proximal operator given by:

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b).

-   119. The method according to enumerated example embodiment 118    wherein evaluating the transformed objective function comprises    determining

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   120. The method according to enumerated example embodiment 119    wherein A={circumflex over (f)}∇².-   121. The method according to any one of enumerated example    embodiments 119 and 120 wherein b=1−I_(p) ^(k)(v).-   122. The method according to any one of enumerated example    embodiments 119 to 121 wherein α>0 is a regularization parameter.-   123. The method according to any one of enumerated example    embodiments 110 to 122 wherein the method comprises initializing the    phase surface as a constant value.-   124. The method according to any one of enumerated example    embodiments 110 to 123 wherein evaluating the transformed objective    function is performed in parallel for different points.-   125. The method according to enumerated example embodiment 124    wherein evaluating is performed in a graphics processing unit.-   126. The method according to any one of enumerated example    embodiments 110 to 125 comprising displaying the image by    controlling pixels of a phase modulator according to the solution    phase function while illuminating the phase modulator.-   127. The method according to enumerated example embodiment 126    comprising evenly illuminating the phase modulator with collimated    light.-   128. The method according to any one of enumerated example    embodiments 126 to 127 wherein the phase modulator comprises an    array of liquid crystal pixels and the method comprises setting    control signals to the pixels according to the solution phase    function.-   129. The method according to enumerated example embodiment 128    wherein the phase modulator is a LCoS phase modulator.-   130. The method according to enumerated example embodiment 110    wherein the phase modulator is a deformable mirror.-   131. The method according to any one of enumerated example    embodiments 126 to 130 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   132. The method according to any one of enumerated example    embodiments 110 to 131 wherein the phase modulator has a maximum    phase retardation and the method comprises subtracting a multiple of    2π from phase shifts of the phase function that exceed the maximum    phase retardation of the phase modulator.-   133. A method for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   determining a fixed point iteration based on the image data;    -   transforming the fixed point iteration into a frequency space;    -   evaluating the fixed point iteration in the frequency space to        obtain a phase function in the frequency space; and    -   inverse transforming the phase function to obtain a solution        phase function relating the phase of the phase modulator to a        position in two dimensions.-   134. The method according to enumerated example embodiment 133    wherein transforming the fixed point iteration comprises computing a    Fourier transform of the fixed point iteration.-   135. The method according to enumerated example embodiment 134    comprising, before transforming, extending the image data to have    periodic boundary conditions and basing the fixed point iteration on    the extended image data.-   136. The method according to enumerated example embodiment 135    wherein extending the image data comprises making a mirror image of    the image data across each boundary of the image data.-   137. The method according to any one of enumerated example    embodiments 133 to 136 wherein the fixed point iteration is a least    squares fixed point iteration.-   138. The method according to any one of enumerated example    embodiments 133 to 137 wherein the fixed point iteration includes a    cost for deviating from an input argument.-   139. The method according to any one of enumerated example    embodiments 133 to 138 wherein the method is performed iteratively    and in each of a plurality of iterations the input argument for the    fixed point iteration is the solution phase function for a previous    iteration.-   140. The method according to enumerated example embodiment 139    comprising caching a Fourier transform of the solution phase    function for the previous iteration and applying the cached Fourier    transform of the solution phase function in a current iteration.-   141. The method according to any one of enumerated example    embodiments 133 to 140 wherein the fixed point iteration comprises a    proximal operator given by:

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b).

-   142. The method according to enumerated example embodiment 141    wherein evaluating the transformed fixed point iteration comprises    determining

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   143. The method according to enumerated example embodiment 142    wherein A={circumflex over (f)}∇².-   144. The method according to any one of enumerated example    embodiments 142 and 143 wherein b=1−I_(p) ^(k)(v).-   145. The method according to any one of enumerated example    embodiments 143 to 144 wherein α>0 is a regularization parameter.-   146. The method according to any one of enumerated example    embodiments 133 to 145 wherein the method comprises initializing the    phase surface as a constant value.-   147. The method according to any one of enumerated example    embodiments 133 to 146 wherein evaluating the transformed fixed    point iteration is performed in parallel for different points.-   148. The method according to enumerated example embodiment 147    wherein evaluating is performed in a graphics processing unit.-   149. The method according to any one of enumerated example    embodiments 133 to 148 comprising displaying the image by    controlling pixels of a phase modulator according to the solution    phase function while illuminating the phase modulator.-   150. The method according to enumerated example embodiment 17    comprising evenly illuminating the phase modulator with collimated    light.-   151. The method according to any one of enumerated example    embodiments 149 to 150 wherein the phase modulator comprises an    array of liquid crystal pixels and the method comprises setting    control signals to the pixels according to the solution phase    function.-   152. The method according to enumerated example embodiment 151    wherein the phase modulator is a LCoS phase modulator.-   153. The method according to enumerated example embodiment 151    wherein the phase modulator is a deformable mirror.-   154. The method according to any one of enumerated example    embodiments 149 to 153 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   155. The method according to any one of enumerated example    embodiments 133 to 154 wherein the phase modulator has a maximum    phase retardation and the method comprises subtracting a multiple of    2π from phase shifts of the phase function that exceed the maximum    phase retardation of the phase modulator.-   156. Apparatus for generating control values for a phase modulator    from image data defining a target light pattern, the apparatus    comprising a data processor in communication with the phase    modulator, the data processor configured to:    -   establish a mapping between points in the light pattern and        corresponding points on the phase modulator;    -   using the mapping, derive a phase function, p, that includes the        control values by mapping the target light pattern into a        coordinate space of the phase modulator; and    -   process the mapped target light pattern in the coordinate space        of the phase modulator.-   157. Apparatus according to enumerated example embodiment 156    wherein the data processor being configured to process the mapped    target light pattern comprises the data processor being configured    to optimize a trial phase function based on comparisons of    intensities in the mapped target light pattern at the points on the    phase modulator to corresponding optical properties of the phase    function in neighborhoods of the points.-   158. Apparatus according to enumerated example embodiment 157    wherein the corresponding optical properties comprise    magnifications.-   159. Apparatus according to any one of enumerated example    embodiments 157 and 158 wherein the data processor is configured to    determine the optical properties based on a Laplacian of the phase    function at the corresponding points.-   160. Apparatus according to enumerated example embodiment 159    wherein the data processor is configured to determine the Laplacian    of the phase function using a discrete Laplacian operator.-   161. Apparatus for displaying video data, the video data specifying    video frames for display at a frame rate, the apparatus comprising a    data processor configured to:    -   in real time, process the video data to yield a sequence of        phase-modulator control signals at the frame rate,    -   apply the phase modulator control signals to an illuminated        two-dimensional spatial phase modulator, and    -   control the spatial phase modulator to direct resulting        phase-modulated light to a viewing area.-   162. Apparatus according to enumerated example embodiment 161    wherein the data processor is configured to control a spatial light    modulator in a path of the phase-modulated light to    amplitude-modulate the phase-modulated light.-   163. Apparatus according to enumerated example embodiment 162    wherein the data processor is configured to compute a blur in the    phase-modulated light and control the spatial light modulator to    reduce the blur.-   164. Apparatus according to any one of enumerated example    embodiments 161 to 163 wherein the data processor being configured    to process the video data comprises the data processor being    configured to:    -   establish a mapping between points in the light pattern and        corresponding points on the light modulator;    -   using the mapping, derive a phase function, p, that includes the        control values by mapping the target light pattern into a        coordinate space of the phase modulator; and    -   process the mapped target light pattern in the coordinate space        of the phase modulator.-   165. Apparatus according to enumerated example embodiment 164    wherein the data processor being configured to process the mapped    target light pattern comprises the data processor being configured    to optimize a trial phase function based on comparisons of    intensities in the mapped target light pattern at the points on the    phase modulator to corresponding optical properties of the phase    function in neighborhoods of the points.-   166. Apparatus according to enumerated example embodiment 165    wherein the corresponding optical properties comprise    magnifications.-   167. Apparatus according to enumerated example embodiment 165 or 166    wherein the data processor is configured to determine the optical    properties based on a Laplacian of the phase function at the    corresponding points.-   168. Apparatus according to enumerated example embodiment 167    wherein the data processor is configured to determine the Laplacian    of the phase function using a discrete Laplacian operator.-   169. Apparatus according to any one of enumerated example    embodiments 161 to 165 wherein the data processor is configured to    process the video data in a frequency domain.-   170. Apparatus according to enumerated example embodiment 169    wherein the data processor being configured to process the video    data comprises the data processor being configured to:    -   generate an optimization function;    -   generate the phase function in the frequency domain by        performing a Fourier transform on the optimization function; and    -   perform an inverse Fourier transformation on the phase function        in the frequency domain.-   171. Apparatus according to enumerated example embodiment 170    comprising hardware configured to perform the Fourier transform and    the data processor is configured to control the hardware to perform    the Fourier transform.-   172. Apparatus according to any one of enumerated example    embodiments 161 to 165 and 169 to 171 wherein the phase modulator    has a maximum phase retardation and the data processor is configured    to subtract a multiple of 27 from phase shifts of the phase function    that exceed the maximum phase retardation of the phase modulator.-   173. Apparatus for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   determining an objective function based on the image data;    -   transforming the objective function into a frequency space;    -   minimizing the transformed objective function in the frequency        space to obtain a phase function in the frequency space; and    -   inverse transform the phase function to obtain a solution phase        function relating the phase of the phase modulator to a position        in two dimensions.-   174. Apparatus according to enumerated example embodiment 173    wherein the data processor being configured to transform the    objective function comprises the data processor being configured to    compute a Fourier transform of the objective function.-   175. Apparatus according to enumerated example embodiment 174    wherein the data processor is configured to, before transforming,    extend the image data to have periodic boundary conditions and base    the objective function on the extended image data.-   176. Apparatus according to enumerated example embodiment 175    wherein the data processor being configured to extend the image data    comprises the data processor being configured to make a mirror image    of the image data across each boundary of the image data.-   177. Apparatus according to any one of enumerated example    embodiments 173 to 176 wherein the objective function is a least    squares objective function.-   178. Apparatus according to any one of enumerated example    embodiments 173 to 177 wherein the objective function includes a    cost for deviating from an input argument.-   179. Apparatus according to any one of enumerated example    embodiments 173 to 178 wherein the data processor is configured to    iteratively determine the objective function, transform the    objective function, and evaluate the transformed objective function,    and in each of a plurality of iterations the input argument for the    objective function is the solution phase function for a previous    iteration.-   180. Apparatus according to enumerated example embodiment 179    wherein the data processor is configured to cache a Fourier    transform of the solution phase function for the previous iteration    and apply the cached Fourier transform of the solution phase    function in a current iteration.-   181. Apparatus according to any one of enumerated example    embodiments 173 to 180 wherein the objective function comprises a    proximal operator given by:

prox_(γF)(q)=(γ+A ^(T) A)−1(γq+A ^(T) b).

-   182. Apparatus according to enumerated example embodiment 181    wherein the data processor being configured to evaluate the    transformed objective function comprises the data processor being    configured to determine

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   183. Apparatus according to enumerated example embodiment 182    wherein A={circumflex over (f)}∇².-   184. Apparatus according to enumerated example embodiment 182 or 183    wherein b=1−I_(p) ^(k)(v).-   185. Apparatus according to any one of enumerated example    embodiments 182 to 184 wherein α>0 is a regularization parameter.-   186. Apparatus according to any one of enumerated example    embodiments 173 to 185 wherein the data processor is configured to    initializing the phase surface as a constant value.-   187. Apparatus according to any one of enumerated example    embodiments 173 to 186 wherein the data processor is configured to    evaluate the transformed objective function in parallel for    different points.-   188. Apparatus according to enumerated example embodiment 187    wherein the data processor comprises a graphics processing unit and    the graphics processing unit evaluates the transformed objective    function.-   189. Apparatus according to any one of enumerated example    embodiments 173 to 188 comprising the phase modulator and a light    source for projecting light on the phase modulator, the data    processor configured to control the phase modulator to display the    image by controlling pixels of the phase modulator according to the    solution phase function while controlling the light source to    illuminate the phase modulator.-   190. Apparatus according to enumerated example embodiment 189    wherein the light source is configured to evenly illuminate the    phase modulator with collimated light.-   191. Apparatus according to any one of enumerated example    embodiments 189 to 190 wherein the phase modulator comprises an    array of liquid crystal pixels and the data processor is configured    to set control signals to the pixels according to the solution phase    function.-   192. Apparatus according to enumerated example embodiment 191    wherein the phase modulator is a LCoS phase modulator.-   193. Apparatus according to enumerated example embodiment 19 wherein    the phase modulator is a deformable mirror.-   194. Apparatus according to any one of enumerated example    embodiments 189 to 193 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   195. Apparatus according to any one of enumerated example    embodiments 173 to 194 wherein the phase modulator has a maximum    phase retardation and the processor is configured to subtract a    multiple of 2π from any phase shifts of the phase function that    exceed the maximum phase retardation of the phase modulator.-   196. Apparatus for controlling a phase modulator to display an image    defined by image data, the apparatus comprising a data processor in    communication with the phase modulator, the data processor    configured to:    -   determine a fixed point iteration based on the image data;    -   transform the fixed point iteration into a frequency space;    -   evaluate the fixed point iteration in the frequency space to        obtain a phase function in the frequency space; and    -   inverse transform the phase function to obtain a solution phase        function relating the phase of the phase modulator to a position        in two dimensions.-   197. Apparatus according to enumerated example embodiment 196    wherein the data processor being configured to transform the fixed    point iteration comprises the data processor being configured to    compute a Fourier transform of the fixed point iteration.-   198. Apparatus according to enumerated example embodiment 197    wherein the data processor is configured to, before transforming,    extend the image data to have periodic boundary conditions and base    the fixed point iteration on the extended image data.-   199. Apparatus according to enumerated example embodiment 198    wherein the data processor being configured to extend the image data    comprises the data processor being configured to make a mirror image    of the image data across each boundary of the image data.-   200. Apparatus according to any one of enumerated example    embodiments 196 to 199 wherein the fixed point iteration is a least    squares fixed point iteration.-   201. Apparatus according to any one of enumerated example    embodiments 196 to 200 wherein the fixed point iteration includes a    cost for deviating from an input argument.-   202. Apparatus according to any one of enumerated example    embodiments 196 to 201 wherein the data processor is configured to    iteratively determine the fixed point iteration, transform the fixed    point iteration, and evaluate the transformed fixed point iteration,    and in each of a plurality of iterations the input argument for the    fixed point iteration is the solution phase function for a previous    iteration.-   203. Apparatus according to enumerated example embodiment 202    wherein the data processor is configured to cache a Fourier    transform of the solution phase function for the previous iteration    and apply the cached Fourier transform of the solution phase    function in a current iteration.-   204. Apparatus according to any one of enumerated example    embodiments 196 to 203 wherein the fixed point iteration comprises a    proximal operator given by:

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b).

-   205. Apparatus according to enumerated example embodiment 204    wherein the data processor being configured to evaluate the    transformed fixed point iteration comprises the data processor being    configured to determine

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   206. Apparatus according to enumerated example embodiment 205    wherein A={circumflex over (f)}∇².-   207. Apparatus according to enumerated example embodiment 205 or 206    wherein b=1−I_(p) ^(k)(v).-   208. Apparatus according to any one of enumerated example    embodiments 205 to 207 wherein α>0 is a regularization parameter.-   209. Apparatus according to any one of enumerated example    embodiments 196 to 208 wherein the data processor is configured to    initializing the phase surface as a constant value.-   210. Apparatus according to any one of enumerated example    embodiments 196 to 209 wherein the data processor is configured to    evaluate the transformed fixed point iteration in parallel for    different points.-   211. Apparatus according to enumerated example embodiment 210    wherein the data processor comprises a graphics processing unit and    the graphics processing unit evaluates the transformed fixed point    iteration.-   212. Apparatus according to any one of enumerated example    embodiments 196 to 211 comprising the phase modulator and a light    source for projecting light on the phase modulator, the data    processor configured to control the phase modulator to display the    image by controlling pixels of the phase modulator according to the    solution phase function while controlling the light source to    illuminate the phase modulator.-   213. Apparatus according to enumerated example embodiment 212    wherein the light source is configured to evenly illuminate the    phase modulator with collimated light.-   214. Apparatus according to any one of enumerated example    embodiments 212 to 213 wherein the phase modulator comprises an    array of liquid crystal pixels and the data processor is configured    to set control signals to the pixels according to the solution phase    function.-   215. Apparatus according to enumerated example embodiment 214    wherein the phase modulator is a LCoS phase modulator.-   216. Apparatus according to enumerated example embodiment 214    wherein the phase modulator is a deformable mirror.-   217. Apparatus according to any one of enumerated example    embodiments 212 to 216 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   218. Apparatus according to any one of enumerated example    embodiments 1 to 22 wherein the phase modulator has a maximum phase    retardation and the processor is configured to subtract a multiple    of 2π from any phase shifts of the phase function that exceed the    maximum phase retardation of the phase modulator.-   219. A method for controlling a phase modulator to display an image    defined by image data, the method comprising:    -   determining a proximal operator of an objective function based        on the image data;    -   transforming the proximal operator into a frequency space;    -   evaluating the transformed proximal operator in the frequency        space to obtain a phase function in the frequency space and        inverse transforming the phase function to obtain a solution        phase function relating a phase of the phase modulator to        position in two dimensions.-   220. The method according to enumerated example embodiment 219    wherein transforming the proximal operator comprises computing a    Fourier transform of the proximal operator.-   221. The method according to enumerated example embodiment 220    comprising, before transforming, extending the image data to have    periodic boundary conditions and basing the proximal operator on the    extended image data.-   222. The method according to enumerated example embodiment 221    wherein extending the image data comprises making a mirror image of    the image data across each boundary of the image data.-   223. The method according to any one of enumerated example    embodiments 219 to 222 wherein the objective function is a least    squares objective function.-   224. The method according to any one of enumerated example    embodiments 219 to 223 wherein the proximal operator includes a cost    for deviating from an input argument.-   225. The method according to any one of enumerated example    embodiments 219 to 224 wherein the method is performed iteratively    and in each of a plurality of iterations the input argument for the    proximal operator is the solution phase function for a previous    iteration.-   226. The method according to enumerated example embodiment 225    comprising caching a Fourier transform of the solution phase    function for the previous iteration and applying the cached Fourier    transform of the solution phase function in a current iteration.-   227. The method according to any one of enumerated example    embodiments 219 to 226 wherein the proximal operator is given by:

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b).

-   228. The method according to enumerated example embodiment 227    wherein evaluating the transformed proximal operator comprises    determining

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   229. The method according to enumerated example embodiment 226    wherein A={circumflex over (f)}∇².-   230. The method according to any one of enumerated example    embodiments 228 and 229 wherein b=1−I_(p) ^(k)(v).-   231. The method according to any one of enumerated example    embodiments 228 to 230 wherein α>0 is a regularization parameter.-   232. The method according to any one of enumerated example    embodiments 219 to 231 wherein the method comprises initializing the    phase surface as a constant value.-   233. The method according to any one of enumerated example    embodiments 219 to 232 wherein evaluating the transformed proximal    operator is performed in parallel for different points.-   234. The method according to enumerated example embodiment 233    wherein evaluating is performed in a graphics processing unit.-   235. The method according to any one of enumerated example    embodiments 219 to 234 comprising displaying the image by    controlling pixels of a phase modulator according to the solution    phase function while illuminating the phase modulator.-   236. The method according to enumerated example embodiment 235    comprising evenly illuminating the phase modulator with collimated    light.-   237. The method according to any one of enumerated example    embodiments 235 to 236 wherein the phase modulator comprises an    array of liquid crystal pixels and the method comprises setting    control signals to the pixels according to the solution phase    function.-   238. The method according to enumerated example embodiment 237    wherein the phase modulator is a LCoS phase modulator.-   239. The method according to enumerated example embodiment 237    wherein the phase modulator is a deformable mirror.-   240. The method according to any one of enumerated example    embodiments 235 to 239 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   241. The method according to any one of enumerated example    embodiments 219 to 240 wherein the phase modulator has a maximum    phase retardation and the method comprises subtracting a multiple of    2π from phase shifts of the phase function that exceed the maximum    phase retardation of the phase modulator.-   242. Apparatus for controlling a phase modulator to display an image    defined by image data, the apparatus comprising a data processor in    communication with the phase modulator, the data processor    configured to:    -   determine a proximal operator of an objective function based on        the image data;    -   transform the proximal operator into a frequency space;    -   evaluate the transformed proximal operator in the frequency        space to obtain a phase function in the frequency space and        inverse transform the phase function to obtain a solution phase        function relating a phase of the phase modulator to position in        two dimensions.-   243. Apparatus according to enumerated example embodiment 242    wherein the data processor being configured to transform the    proximal operator comprises the data processor being configured to    compute a Fourier transform of the proximal operator.-   244. Apparatus according to enumerated example embodiment 243    wherein the data processor is configured to, before transforming,    extend the image data to have periodic boundary conditions and base    the proximal operator on the extended image data.-   245. Apparatus according to enumerated example embodiment 244    wherein the data processor being configured to extend the image data    comprises the data processor being configured to make a mirror image    of the image data across each boundary of the image data.-   246. Apparatus according to any one of enumerated example    embodiments 242 to 245 wherein the objective function is a least    squares objective function.-   247. Apparatus according to any one of enumerated example    embodiments 242 to 246 wherein the proximal operator includes a cost    for deviating from an input argument.-   248. Apparatus according to any one of enumerated example    embodiments 242 to 247 wherein the data processor is configured to    iteratively determine the proximal operator, transform the proximal    operator, and evaluate the transformed proximal operator, and in    each of a plurality of iterations the input argument for the    proximal operator is the solution phase function for a previous    iteration.-   249. Apparatus according to enumerated example embodiment 248    wherein the data processor is configured to cache a Fourier    transform of the solution phase function for the previous iteration    and apply the cached Fourier transform of the solution phase    function in a current iteration.-   250. Apparatus according to any one of enumerated example    embodiments 242 to 249 wherein the proximal operator is given by:

prox_(γF)(q)=(γ+A ^(T) A)⁻¹(γq+A ^(T) b).

-   251. Apparatus according to enumerated example embodiment 250    wherein the data processor being configured to evaluate the    transformed proximal operator comprises the data processor being    configured to determine

$\left( \frac{{{\mathcal{F}(b)}{\mathcal{F}(A)}^{*}} + {\gamma \; {\mathcal{F}(q)}}}{{\left( {1 + \alpha} \right){\mathcal{F}(A)}^{2}} + \gamma} \right).$

-   252. Apparatus according to enumerated example embodiment 251    wherein A={circumflex over (f)}∇².-   253. Apparatus according to enumerated example embodiment 251 or 252    wherein b=1−I_(p) ^(k)(v).-   254. Apparatus according to any one of enumerated example    embodiments 251 to 253 wherein α>0 is a regularization parameter.-   255. Apparatus according to any one of enumerated example    embodiments 242 to 254 wherein the data processor is configured to    initializing the phase surface as a constant value.-   256. Apparatus according to any one of enumerated example    embodiments 242 to 255 wherein the data processor is configured to    evaluate the transformed proximal operator in parallel for different    points.-   257. Apparatus according to enumerated example embodiment 256    wherein the data processor comprises a graphics processing unit and    the graphics processing unit evaluates the transformed proximal    operator.-   258. Apparatus according to any one of enumerated example    embodiments 242 to 257 comprising the phase modulator and a light    source for projecting light on the phase modulator, the data    processor configured to control the phase modulator to display the    image by controlling pixels of the phase modulator according to the    solution phase function while controlling the light source to    illuminate the phase modulator.-   259. Apparatus according to enumerated example embodiment 258    wherein the light source is configured to evenly illuminate the    phase modulator with collimated light.-   260. Apparatus according to any one of enumerated example    embodiments 258 to 259 wherein the phase modulator comprises an    array of liquid crystal pixels and the data processor is configured    to set control signals to the pixels according to the solution phase    function.-   261. Apparatus according to enumerated example embodiment 260    wherein the phase modulator is a LCoS phase modulator.-   262. Apparatus according to enumerated example embodiment 260    wherein the phase modulator is a deformable mirror.-   263. Apparatus according to any one of enumerated example    embodiments 258 to 262 wherein a maximum numerical aperture for    points on the phase modulator is 0.21 or less.-   264. Apparatus according to any one of enumerated example    embodiments 242 to 264 wherein the phase modulator has a maximum    phase retardation and the processor is configured to subtract a    multiple of 2π from any phase shifts of the phase function that    exceed the maximum phase retardation of the phase modulator.-   265. A computer-readable medium including computer-readable software    instructions configured to cause a data processor to perform the    method of any one of the above method enumerated example    embodiments.-   266. Apparatus having any new and inventive feature, combination of    features, or sub-combination of features as described herein.-   267. Methods having any new and inventive steps, acts, combination    of steps and/or acts or sub-combination of steps and/or acts as    described herein.

It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such modifications,permutations, additions, omissions, and sub-combinations as mayreasonably be inferred. The scope of the claims should not be limited bythe preferred embodiments set forth in the examples, but should be giventhe broadest interpretation consistent with the description as a whole.

1.-267. (canceled)
 268. A method for displaying video data, the videodata specifying video frames for display at a frame rate, the methodcomprising: in real time, processing the video data to yield a sequenceof phase-modulator control signals at the frame rate, applying the phasemodulator control signals to an illuminated two-dimensional spatialphase modulator, and directing resulting phase-modulated light to aviewing area.
 269. The method according to claim 268 comprising furtheramplitude-modulating the phase-modulated light.
 270. The methodaccording to claim 269 wherein further amplitude-modulating thephase-modulated light comprises controlling a spatial light modulator ina path of the phase-modulated light.
 271. The method according to claim270 comprising computing a blur in the phase-modulated light andcontrolling the spatial light modulator to reduce the blur.
 272. Themethod according to claim 268 wherein processing the video datacomprises: establishing a mapping between points in the light patternand corresponding points on the light modulator; using the mapping,deriving a phase function, p, that includes the control values bymapping the target light pattern into a coordinate space of the phasemodulator; and processing the mapped target light pattern in thecoordinate space of the phase modulator.
 273. The method according toclaim 272 wherein processing the mapped target light pattern comprisesoptimizing a trial phase function based on comparisons of intensities inthe mapped target light pattern at the points on the phase modulator tocorresponding optical properties of the phase function in neighborhoodsof the points.
 274. The method according to claim 273 wherein thecorresponding optical properties comprise magnifications.
 275. Themethod according to claim 273 comprising determining the opticalproperties based on a Laplacian of the phase function at thecorresponding points.
 276. The method according to claim 275 comprisingdetermining the Laplacian of the phase function using a discreteLaplacian operator.
 277. The method according to claim 268 whereinprocessing the video data is performed in a frequency domain.
 278. Themethod according to claim 277 wherein processing the video datacomprises generating an optimization function; performing a Fouriertransform on the optimization function generating the phase function inthe frequency domain and performing an inverse Fourier transformation onthe phase function.
 279. The method according to claim 278 comprisingperforming the Fourier transform in hardware configured to perform theFourier transform.
 280. The method according to claim 277 wherein thephase modulator has a maximum phase retardation and the method comprisessubtracting a multiple of 2π from phase shifts of the phase functionthat exceed the maximum phase retardation of the phase modulator. 281.Apparatus for displaying video data, the video data specifying videoframes for display at a frame rate, the apparatus comprising a dataprocessor configured to: in real time, process the video data to yield asequence of phase-modulator control signals at the frame rate, apply thephase modulator control signals to an illuminated two-dimensionalspatial phase modulator, and control the spatial phase modulator todirect resulting phase-modulated light to a viewing area.
 282. Apparatusaccording to claim 281 wherein the data processor is configured tocontrol a spatial light modulator in a path of the phase-modulated lightto amplitude-modulate the phase-modulated light.
 283. Apparatusaccording to claim 282 wherein the data processor is configured tocompute a blur in the phase-modulated light and control the spatiallight modulator to reduce the blur.
 284. Apparatus according to claim281 wherein the data processor being configured to process the videodata comprises the data processor being configured to: establish amapping between points in the light pattern and corresponding points onthe light modulator; using the mapping, derive a phase function, p, thatincludes the control values by mapping the target light pattern into acoordinate space of the phase modulator; and process the mapped targetlight pattern in the coordinate space of the phase modulator. 285.Apparatus according to claim 284 wherein the data processor beingconfigured to process the mapped target light pattern comprises the dataprocessor being configured to optimize a trial phase function based oncomparisons of intensities in the mapped target light pattern at thepoints on the phase modulator to corresponding optical properties of thephase function in neighborhoods of the points.
 286. Apparatus accordingto claim 285 wherein the corresponding optical properties comprisemagnifications.
 287. Apparatus according to claim 285 wherein the dataprocessor is configured to determine the optical properties based on aLaplacian of the phase function at the corresponding points. 288.Apparatus according to claim 287 wherein the data processor isconfigured to determine the Laplacian of the phase function using adiscrete Laplacian operator.
 289. Apparatus according claim 281 whereinthe data processor is configured to process the video data in afrequency domain.
 290. Apparatus according to claim 289 wherein the dataprocessor being configured to process the video data comprises the dataprocessor being configured to: generate an optimization function;generate the phase function in the frequency domain by performing aFourier transform on the optimization function; and perform an inverseFourier transformation on the phase function in the frequency domain.291. Apparatus according to claim 290 comprising hardware configured toperform the Fourier transform and the data processor is configured tocontrol the hardware to perform the Fourier transform.
 292. Apparatusaccording to claim 289 wherein the phase modulator has a maximum phaseretardation and the data processor is configured to subtract a multipleof 2π from phase shifts of the phase function that exceed the maximumphase retardation of the phase modulator.